Max level shown:
1 [ May 8 03:32:09 Enabled. ]
2 [ May 8 03:32:09 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ May 8 03:32:09 Method "start" exited with status 0. ]
4 [ May 8 03:32:09 Rereading configuration. ]
5 [ May 8 03:32:09 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-05-08T03:32:09.661ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:163
82024-05-08T03:32:09.661ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-05-08T03:32:09.714ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-05-08T03:32:09.714ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-05-08T03:32:09.825ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-05-08T03:32:09.835ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-05-08T03:32:09.842ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-05-08T03:32:10.447ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:9fb2::, len: 64 }
152024-05-08T03:32:10.587ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:119
162024-05-08T03:32:10.587ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:131
172024-05-08T03:32:10.587ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:141
182024-05-08T03:32:10.587ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:162 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-05-08T03:32:10.587ZINFOSledAgent (DumpSetup-archival): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:291
202024-05-08T03:32:10.587ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
212024-05-08T03:32:10.587ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:625
222024-05-08T03:32:10.590ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:535
232024-05-08T03:32:10.590ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:674
242024-05-08T03:32:10.590ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:176
252024-05-08T03:32:10.590ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:733
262024-05-08T03:32:10.590ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_0.vdev
272024-05-08T03:32:10.590ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = None
282024-05-08T03:32:10.590ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }), tx: ... } file = sled-storage/src/manager.rs:381
292024-05-08T03:32:10.590ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
302024-05-08T03:32:10.590ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } path = /scratch/m2_0.vdev
312024-05-08T03:32:10.602ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:304
322024-05-08T03:32:10.602ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool)
332024-05-08T03:32:10.698ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal }
342024-05-08T03:32:10.901ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
352024-05-08T03:32:10.901ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_1.vdev
362024-05-08T03:32:10.901ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
372024-05-08T03:32:10.901ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
382024-05-08T03:32:10.901ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }), tx: ... } file = sled-storage/src/manager.rs:381
392024-05-08T03:32:10.901ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
402024-05-08T03:32:10.901ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } path = /scratch/m2_1.vdev
412024-05-08T03:32:10.909ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:304
422024-05-08T03:32:10.909ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool)
432024-05-08T03:32:10.999ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal }
442024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_0.vdev
452024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
462024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }), tx: ... } file = sled-storage/src/manager.rs:381
472024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
482024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_1.vdev
492024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }), tx: ... } file = sled-storage/src/manager.rs:381
502024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
512024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
522024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_2.vdev
532024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
542024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }), tx: ... } file = sled-storage/src/manager.rs:381
552024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
562024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_3.vdev
572024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
582024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }), tx: ... } file = sled-storage/src/manager.rs:381
592024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
602024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_4.vdev
612024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }), tx: ... } file = sled-storage/src/manager.rs:381
622024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
632024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
642024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_5.vdev
652024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }), tx: ... } file = sled-storage/src/manager.rs:381
662024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
672024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
682024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_6.vdev
692024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }), tx: ... } file = sled-storage/src/manager.rs:381
702024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
712024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
722024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_7.vdev
732024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }), tx: ... } file = sled-storage/src/manager.rs:381
742024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
752024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
762024-05-08T03:32:11.201ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_8.vdev
772024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }), tx: ... } file = sled-storage/src/manager.rs:381
782024-05-08T03:32:11.201ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
792024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
802024-05-08T03:32:11.201ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:91
812024-05-08T03:32:11.201ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-agent/src/long_running_tasks.rs:93
822024-05-08T03:32:11.201ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
832024-05-08T03:32:11.201ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
842024-05-08T03:32:11.201ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:200
852024-05-08T03:32:11.201ZINFOSledAgent (bootstore): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
862024-05-08T03:32:11.201ZINFOSledAgent (bootstore): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
872024-05-08T03:32:11.201ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
882024-05-08T03:32:11.201ZINFOSledAgent (bootstore): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
892024-05-08T03:32:11.201ZINFOSledAgent (bootstore): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
902024-05-08T03:32:11.201ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
912024-05-08T03:32:11.201ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:205
922024-05-08T03:32:11.202ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:220
932024-05-08T03:32:11.212ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:711
942024-05-08T03:32:11.212ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 10728, tv_nsec: 5838538 } time_to_next_cleanup = 599.99993764s
952024-05-08T03:32:11.213ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3617
962024-05-08T03:32:11.213ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
972024-05-08T03:32:11.215ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/sled-agent-request.json file = common/src/ledger.rs:188
982024-05-08T03:32:11.215ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/sled-agent-request.json file = common/src/ledger.rs:188
992024-05-08T03:32:11.215ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:195 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1002024-05-08T03:32:11.215ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1012024-05-08T03:32:11.216ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV1 { routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.1.100, }, ], addresses: [ V4( Ipv4Network { addr: 10.151.1.174, prefix: 24, }, ), ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, } file = sled-agent/src/rack_setup/service.rs:979
1022024-05-08T03:32:11.216ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1032024-05-08T03:32:11.216ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1042024-05-08T03:32:11.216ZINFOSledAgent (RSS): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1052024-05-08T03:32:11.216ZINFOSledAgent (RSS): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1062024-05-08T03:32:11.216ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1039
1072024-05-08T03:32:11.216ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1082024-05-08T03:32:11.216ZINFOSledAgent (RSS): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/rss-sled-plan.json file = common/src/ledger.rs:188
1092024-05-08T03:32:11.216ZINFOSledAgent (RSS): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/rss-sled-plan.json file = common/src/ledger.rs:188
1102024-05-08T03:32:11.216ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1080
1112024-05-08T03:32:11.216ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:118
1122024-05-08T03:32:11.216ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:91
1132024-05-08T03:32:11.216ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1142024-05-08T03:32:11.216ZINFOSledAgent (RSS): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1152024-05-08T03:32:11.217ZINFOSledAgent (RSS): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1162024-05-08T03:32:11.217ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:138
1172024-05-08T03:32:11.217ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1113
1182024-05-08T03:32:11.217ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1192024-05-08T03:32:11.217ZINFOSledAgent (bootstore): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1202024-05-08T03:32:11.217ZINFOSledAgent (bootstore): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1212024-05-08T03:32:11.217ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1222024-05-08T03:32:11.217ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 411336b5-dbaf-48b1-93b3-a95d20f0f732, rack_id: 76e0fa5a-5dcb-4021-ab9c-640c3670ef01, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1232024-05-08T03:32:11.217ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:49602
1242024-05-08T03:32:11.217ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 411336b5-dbaf-48b1-93b3-a95d20f0f732, rack_id: 76e0fa5a-5dcb-4021-ab9c-640c3670ef01, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } file = sled-agent/src/bootstrap/server.rs:360
1252024-05-08T03:32:11.217ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:375
1262024-05-08T03:32:11.217ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1272024-05-08T03:32:11.217ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:381
1282024-05-08T03:32:11.217ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:530
1292024-05-08T03:32:11.217ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1302024-05-08T03:32:11.218ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1312024-05-08T03:32:11.218ZINFOSledAgent (StorageManager): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1322024-05-08T03:32:11.218ZINFOSledAgent (StorageManager): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1332024-05-08T03:32:11.218ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:519
1342024-05-08T03:32:11.218ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:547
1352024-05-08T03:32:11.220ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3849
1362024-05-08T03:32:11.228ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
1372024-05-08T03:32:11.237ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
1382024-05-08T03:32:11.249ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
1392024-05-08T03:32:11.259ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
1402024-05-08T03:32:11.268ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
1412024-05-08T03:32:11.276ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
1422024-05-08T03:32:11.284ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
1432024-05-08T03:32:11.293ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
1442024-05-08T03:32:11.302ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
1452024-05-08T03:32:11.302ZINFOSledAgent (StorageManager): No disks to be automatically managed file = sled-storage/src/manager.rs:558
1462024-05-08T03:32:11.302ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1472024-05-08T03:32:11.302ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:370
1482024-05-08T03:32:11.302ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
1492024-05-08T03:32:11.302ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:449 result = DisksManagementResult { status: [] }
1502024-05-08T03:32:11.318ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:352 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1512024-05-08T03:32:11.319ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1522024-05-08T03:32:11.319ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:364 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1532024-05-08T03:32:11.319ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1542024-05-08T03:32:11.321ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 111.011805ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1d58111b-934e-4b49-bf27-837dd523bcfd", "content-length": "124", "date": "Wed, 08 May 2024 03:32:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1d58111b-934e-4b49-bf27-837dd523bcfd" } file = clients/ddm-admin-client/src/lib.rs:178
1552024-05-08T03:32:11.323ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:315
1562024-05-08T03:32:11.331ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_80ce766d-f6eb-440d-b6a8-ad5a1011c02b/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1572024-05-08T03:32:11.341ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1582024-05-08T03:32:11.354ZINFOSledAgent: successfully created encrypted zvol: name="oxi_80ce766d-f6eb-440d-b6a8-ad5a1011c02b/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1592024-05-08T03:32:11.354ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_80ce766d-f6eb-440d-b6a8-ad5a1011c02b/swap" file = sled-agent/src/swap_device.rs:96
1602024-05-08T03:32:11.359ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:379 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1612024-05-08T03:32:11.359ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1622024-05-08T03:32:11.359ZINFOSledAgent (BackingFs): Ensuring dataset oxi_80ce766d-f6eb-440d-b6a8-ad5a1011c02b/backing/fmd file = sled-agent/src/backing_fs.rs:136
1632024-05-08T03:32:11.428ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1642024-05-08T03:32:11.434ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1652024-05-08T03:32:11.434ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 105.831014ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "46f64bb9-918f-4910-a8a1-f0b730294684", "content-length": "124", "date": "Wed, 08 May 2024 03:32:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "46f64bb9-918f-4910-a8a1-f0b730294684" } file = clients/ddm-admin-client/src/lib.rs:178
1662024-05-08T03:32:11.540ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1672024-05-08T03:32:11.541ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 215.908375ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b1bc65e8-2181-420e-85ae-980cea2dc6af", "content-length": "124", "date": "Wed, 08 May 2024 03:32:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b1bc65e8-2181-420e-85ae-980cea2dc6af" } file = clients/ddm-admin-client/src/lib.rs:178
1682024-05-08T03:32:11.645ZINFOSledAgent (BackingFs): Mounting oxi_80ce766d-f6eb-440d-b6a8-ad5a1011c02b/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1692024-05-08T03:32:11.657ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1702024-05-08T03:32:11.657ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1712024-05-08T03:32:11.658ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1722024-05-08T03:32:11.658ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1732024-05-08T03:32:11.758ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1742024-05-08T03:32:11.759ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 200.194525ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c4231ce1-42a7-4b4f-be12-0cb0c6c424ab", "content-length": "124", "date": "Wed, 08 May 2024 03:32:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c4231ce1-42a7-4b4f-be12-0cb0c6c424ab" } file = clients/ddm-admin-client/src/lib.rs:178
1752024-05-08T03:32:11.960ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1762024-05-08T03:32:11.961ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 185.930583ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "826ed309-c41d-4555-8190-782c3bab45e9", "content-length": "124", "date": "Wed, 08 May 2024 03:32:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "826ed309-c41d-4555-8190-782c3bab45e9" } file = clients/ddm-admin-client/src/lib.rs:178
1772024-05-08T03:32:12.148ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1782024-05-08T03:32:12.148ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 310.316624ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4dc0183a-f61e-4c0c-9f45-6900db204a5a", "content-length": "124", "date": "Wed, 08 May 2024 03:32:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4dc0183a-f61e-4c0c-9f45-6900db204a5a" } file = clients/ddm-admin-client/src/lib.rs:178
1792024-05-08T03:32:12.404ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1802024-05-08T03:32:12.405ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:195 local_addr = [fd00:1122:3344:101::1]:34994 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1812024-05-08T03:32:12.405ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1822024-05-08T03:32:12.405ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:34994 file = oximeter/producer/src/lib.rs:284 interval = 30s producer_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 route = //411336b5-dbaf-48b1-93b3-a95d20f0f732 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1832024-05-08T03:32:12.411ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 176.285704ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1842024-05-08T03:32:12.450ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1852024-05-08T03:32:12.450ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
1862024-05-08T03:32:12.451ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:993 underlay_address = fd00:1122:3344:101::1
1872024-05-08T03:32:12.451ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1882024-05-08T03:32:12.451ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:195 local_addr = [fd00:1122:3344:101::1]:12345
1892024-05-08T03:32:12.451ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:430
1902024-05-08T03:32:12.451ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1912024-05-08T03:32:12.451ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
1922024-05-08T03:32:12.452ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
1932024-05-08T03:32:12.452ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
1942024-05-08T03:32:12.452ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV1 { routes: [RouteConfig { destination: V4(Ipv4Network { addr: 0.0.0.0, prefix: 0 }), nexthop: 10.151.1.100 }], addresses: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
1952024-05-08T03:32:12.452ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3617
1962024-05-08T03:32:12.452ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:49602
1972024-05-08T03:32:12.628ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
1982024-05-08T03:32:12.628ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1992024-05-08T03:32:12.628ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2002024-05-08T03:32:12.628ZINFOSledAgent (RSS): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2012024-05-08T03:32:12.628ZINFOSledAgent (RSS): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2022024-05-08T03:32:12.629ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2032024-05-08T03:32:12.629ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2042024-05-08T03:32:12.629ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
2052024-05-08T03:32:12.630ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 585.568427ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2062024-05-08T03:32:12.634ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2072024-05-08T03:32:12.661ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50126
2082024-05-08T03:32:12.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2092024-05-08T03:32:12.661ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 224 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50126 req_id = c1bd27ec-071b-43fb-ace3-bf6c50dc6e3b response_code = 200 uri = /inventory
2102024-05-08T03:32:12.692ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53026
2112024-05-08T03:32:12.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53026 req_id = 2f756672-8848-48ed-b081-79df3e4a7c69 response_code = 200 uri = /sled-role
2122024-05-08T03:32:12.693ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2132024-05-08T03:32:12.693ZINFOSledAgent (RSS): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2142024-05-08T03:32:12.693ZINFOSledAgent (RSS): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2152024-05-08T03:32:12.694ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:815
2162024-05-08T03:32:12.725ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: 593fb985-7780-4b02-bf28-d1cdda078929, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, pool_id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool) }, OmicronPhysicalDiskConfig { id: 56c6ff11-976b-414f-9ee5-247de31e94d7, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, pool_id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool) }, OmicronPhysicalDiskConfig { id: 5d9df6f3-acd0-4b34-9a89-678f436a2e72, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, pool_id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool) }, OmicronPhysicalDiskConfig { id: 60318e90-6e2f-49c0-ac8a-bce5d70c9296, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, pool_id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool) }, OmicronPhysicalDiskConfig { id: be1c0728-b856-4b17-885b-3552285d237a, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, pool_id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool) }, OmicronPhysicalDiskConfig { id: f9a17636-0bdc-4226-90b5-bb7633d77062, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, pool_id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool) }, OmicronPhysicalDiskConfig { id: 40f4ad6e-012f-43b5-bbc3-2175572d8df3, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, pool_id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool) }, OmicronPhysicalDiskConfig { id: c6fe1b14-2e40-4c3f-881d-80d18d9669a7, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, pool_id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool) }, OmicronPhysicalDiskConfig { id: 3611d43e-cd95-4f9f-a0b7-065b796797ff, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, pool_id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool) }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:346 sled_address = [fd00:1122:3344:101::1]:12345
2172024-05-08T03:32:12.725ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48070
2182024-05-08T03:32:12.726ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, id: 593fb985-7780-4b02-bf28-d1cdda078929, pool_id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, id: 56c6ff11-976b-414f-9ee5-247de31e94d7, pool_id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, id: 5d9df6f3-acd0-4b34-9a89-678f436a2e72, pool_id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, id: 60318e90-6e2f-49c0-ac8a-bce5d70c9296, pool_id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, id: be1c0728-b856-4b17-885b-3552285d237a, pool_id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, id: f9a17636-0bdc-4226-90b5-bb7633d77062, pool_id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, id: 40f4ad6e-012f-43b5-bbc3-2175572d8df3, pool_id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, id: c6fe1b14-2e40-4c3f-881d-80d18d9669a7, pool_id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, id: 3611d43e-cd95-4f9f-a0b7-065b796797ff, pool_id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:381
2192024-05-08T03:32:12.726ZINFOSledAgent (StorageManager): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2202024-05-08T03:32:12.726ZINFOSledAgent (StorageManager): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2212024-05-08T03:32:12.726ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:681 request = omicron_physical_disks_ensure
2222024-05-08T03:32:12.726ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
2232024-05-08T03:32:12.726ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2242024-05-08T03:32:12.726ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2252024-05-08T03:32:12.726ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } path = /scratch/u2_0.vdev
2262024-05-08T03:32:12.815ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:304
2272024-05-08T03:32:12.815ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool)
2282024-05-08T03:32:13.225ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.159154008s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2292024-05-08T03:32:13.573ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2302024-05-08T03:32:13.756ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }
2312024-05-08T03:32:13.921ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2322024-05-08T03:32:13.921ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2332024-05-08T03:32:13.921ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2342024-05-08T03:32:13.921ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2352024-05-08T03:32:13.921ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2362024-05-08T03:32:13.921ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2372024-05-08T03:32:13.921ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2382024-05-08T03:32:14.505ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.673752662s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2392024-05-08T03:32:14.507ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2402024-05-08T03:32:14.692ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_switch
2412024-05-08T03:32:14.893ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_58eb44a6-f8c0-46ec-bcea-52bf3efa8996/crypt/zone file = sled-storage/src/dataset.rs:395
2422024-05-08T03:32:15.126ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2432024-05-08T03:32:15.126ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2442024-05-08T03:32:15.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 64.313882ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2452024-05-08T03:32:16.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 87.450304ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2462024-05-08T03:32:16.464ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2472024-05-08T03:32:16.464ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 5.09253006s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2482024-05-08T03:32:16.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 211.253261ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2492024-05-08T03:32:16.867ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2502024-05-08T03:32:17.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 281.757535ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2512024-05-08T03:32:17.541ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2522024-05-08T03:32:17.790ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_58eb44a6-f8c0-46ec-bcea-52bf3efa8996 file = sled-storage/src/dataset.rs:480
2532024-05-08T03:32:18.230ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2542024-05-08T03:32:18.230ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2552024-05-08T03:32:18.230ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2562024-05-08T03:32:18.230ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2572024-05-08T03:32:18.230ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } path = /scratch/u2_1.vdev
2582024-05-08T03:32:18.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 425.781846ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2592024-05-08T03:32:18.470ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:304
2602024-05-08T03:32:18.470ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool)
2612024-05-08T03:32:18.684ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2622024-05-08T03:32:18.708ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2632024-05-08T03:32:19.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.386795239s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2642024-05-08T03:32:19.834ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2652024-05-08T03:32:19.849ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }
2662024-05-08T03:32:20.077ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2672024-05-08T03:32:20.077ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2682024-05-08T03:32:20.077ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2692024-05-08T03:32:20.077ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2702024-05-08T03:32:20.077ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2712024-05-08T03:32:20.077ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2722024-05-08T03:32:20.077ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2732024-05-08T03:32:21.061ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2742024-05-08T03:32:21.537ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_888a035b-7ecc-4eff-a005-8001c3d3ccff/crypt/zone file = sled-storage/src/dataset.rs:395
2752024-05-08T03:32:22.064ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2762024-05-08T03:32:22.064ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2772024-05-08T03:32:22.065ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 10.119052907s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:393 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2782024-05-08T03:32:22.803ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2792024-05-08T03:32:22.810ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2802024-05-08T03:32:22.908ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_888a035b-7ecc-4eff-a005-8001c3d3ccff file = sled-storage/src/dataset.rs:480
2812024-05-08T03:32:22.916ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2822024-05-08T03:32:22.916ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2832024-05-08T03:32:22.916ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2842024-05-08T03:32:22.916ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2852024-05-08T03:32:22.916ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } path = /scratch/u2_2.vdev
2862024-05-08T03:32:22.925ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:304
2872024-05-08T03:32:22.925ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool)
2882024-05-08T03:32:23.018ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }
2892024-05-08T03:32:23.026ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2902024-05-08T03:32:23.026ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2912024-05-08T03:32:23.026ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2922024-05-08T03:32:23.026ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2932024-05-08T03:32:23.026ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
2942024-05-08T03:32:23.027ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
2952024-05-08T03:32:23.027ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2962024-05-08T03:32:23.062ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2972024-05-08T03:32:23.067ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
2982024-05-08T03:32:23.070ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1afb2a46-5b47-46d7-9926-14b7e21c6407/crypt/zone file = sled-storage/src/dataset.rs:395
2992024-05-08T03:32:23.192ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1afb2a46-5b47-46d7-9926-14b7e21c6407 file = sled-storage/src/dataset.rs:480
3002024-05-08T03:32:23.200ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3012024-05-08T03:32:23.200ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3022024-05-08T03:32:23.200ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3032024-05-08T03:32:23.200ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3042024-05-08T03:32:23.200ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } path = /scratch/u2_3.vdev
3052024-05-08T03:32:23.210ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:304
3062024-05-08T03:32:23.210ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool)
3072024-05-08T03:32:23.301ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }
3082024-05-08T03:32:23.308ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3092024-05-08T03:32:23.308ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3102024-05-08T03:32:23.308ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3112024-05-08T03:32:23.308ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3122024-05-08T03:32:23.308ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3132024-05-08T03:32:23.309ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3142024-05-08T03:32:23.309ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3152024-05-08T03:32:23.346ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3162024-05-08T03:32:23.353ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7c0211ed-cba6-48d8-8b63-a5f21383d980/crypt/zone file = sled-storage/src/dataset.rs:395
3172024-05-08T03:32:23.466ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7c0211ed-cba6-48d8-8b63-a5f21383d980 file = sled-storage/src/dataset.rs:480
3182024-05-08T03:32:23.474ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3192024-05-08T03:32:23.474ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3202024-05-08T03:32:23.474ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3212024-05-08T03:32:23.474ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3222024-05-08T03:32:23.474ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } path = /scratch/u2_4.vdev
3232024-05-08T03:32:23.483ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:304
3242024-05-08T03:32:23.483ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool)
3252024-05-08T03:32:23.579ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }
3262024-05-08T03:32:23.587ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3272024-05-08T03:32:23.587ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3282024-05-08T03:32:23.587ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3292024-05-08T03:32:23.587ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3302024-05-08T03:32:23.587ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3312024-05-08T03:32:23.587ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3322024-05-08T03:32:23.587ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3332024-05-08T03:32:23.623ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3342024-05-08T03:32:23.631ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_fc47eb1d-9a62-49dd-845d-82b29d12e969/crypt/zone file = sled-storage/src/dataset.rs:395
3352024-05-08T03:32:23.753ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_fc47eb1d-9a62-49dd-845d-82b29d12e969 file = sled-storage/src/dataset.rs:480
3362024-05-08T03:32:23.761ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3372024-05-08T03:32:23.762ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3382024-05-08T03:32:23.762ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3392024-05-08T03:32:23.762ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3402024-05-08T03:32:23.762ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } path = /scratch/u2_5.vdev
3412024-05-08T03:32:23.771ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:304
3422024-05-08T03:32:23.771ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool)
3432024-05-08T03:32:23.864ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }
3442024-05-08T03:32:23.872ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3452024-05-08T03:32:23.872ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3462024-05-08T03:32:23.872ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3472024-05-08T03:32:23.872ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3482024-05-08T03:32:23.872ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3492024-05-08T03:32:23.872ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3502024-05-08T03:32:23.872ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3512024-05-08T03:32:23.910ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3522024-05-08T03:32:23.918ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_62a3be27-a8c4-4992-8657-cb4aacfdb8ab/crypt/zone file = sled-storage/src/dataset.rs:395
3532024-05-08T03:32:24.043ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_62a3be27-a8c4-4992-8657-cb4aacfdb8ab file = sled-storage/src/dataset.rs:480
3542024-05-08T03:32:24.051ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3552024-05-08T03:32:24.051ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3562024-05-08T03:32:24.051ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3572024-05-08T03:32:24.051ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3582024-05-08T03:32:24.051ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } path = /scratch/u2_6.vdev
3592024-05-08T03:32:24.060ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:304
3602024-05-08T03:32:24.061ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool)
3612024-05-08T03:32:24.156ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }
3622024-05-08T03:32:24.165ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3632024-05-08T03:32:24.165ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3642024-05-08T03:32:24.165ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3652024-05-08T03:32:24.165ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3662024-05-08T03:32:24.165ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3672024-05-08T03:32:24.165ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3682024-05-08T03:32:24.165ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3692024-05-08T03:32:24.169ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
3702024-05-08T03:32:24.202ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3712024-05-08T03:32:24.211ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_9edc3ed7-0367-4fc4-87ff-888719d0b682/crypt/zone file = sled-storage/src/dataset.rs:395
3722024-05-08T03:32:24.331ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2303
3732024-05-08T03:32:24.335ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_9edc3ed7-0367-4fc4-87ff-888719d0b682 file = sled-storage/src/dataset.rs:480
3742024-05-08T03:32:24.344ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3752024-05-08T03:32:24.344ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3762024-05-08T03:32:24.344ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3772024-05-08T03:32:24.344ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3782024-05-08T03:32:24.344ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } path = /scratch/u2_7.vdev
3792024-05-08T03:32:24.354ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:304
3802024-05-08T03:32:24.354ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool)
3812024-05-08T03:32:24.452ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }
3822024-05-08T03:32:24.461ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3832024-05-08T03:32:24.461ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3842024-05-08T03:32:24.461ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3852024-05-08T03:32:24.461ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3862024-05-08T03:32:24.461ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3872024-05-08T03:32:24.461ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3882024-05-08T03:32:24.461ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3892024-05-08T03:32:24.497ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3902024-05-08T03:32:24.505ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ed53811e-9a91-4270-bd15-bcb96ef98de1/crypt/zone file = sled-storage/src/dataset.rs:395
3912024-05-08T03:32:24.627ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ed53811e-9a91-4270-bd15-bcb96ef98de1 file = sled-storage/src/dataset.rs:480
3922024-05-08T03:32:24.635ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3932024-05-08T03:32:24.635ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3942024-05-08T03:32:24.635ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3952024-05-08T03:32:24.635ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3962024-05-08T03:32:24.635ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:112 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } path = /scratch/u2_8.vdev
3972024-05-08T03:32:24.645ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:304
3982024-05-08T03:32:24.645ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool)
3992024-05-08T03:32:24.739ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }
4002024-05-08T03:32:24.747ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
4012024-05-08T03:32:24.747ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
4022024-05-08T03:32:24.747ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
4032024-05-08T03:32:24.747ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
4042024-05-08T03:32:24.747ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4052024-05-08T03:32:24.747ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4062024-05-08T03:32:24.748ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4072024-05-08T03:32:24.782ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4082024-05-08T03:32:24.790ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621/crypt/zone file = sled-storage/src/dataset.rs:395
4092024-05-08T03:32:24.810ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
4102024-05-08T03:32:24.909ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 file = sled-storage/src/dataset.rs:480
4112024-05-08T03:32:24.918ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
4122024-05-08T03:32:24.918ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
4132024-05-08T03:32:24.918ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4142024-05-08T03:32:24.918ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 80ce766d-f6eb-440d-b6a8-ad5a1011c02b (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }, zpool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: b3bd066e-6f80-4f80-bc0d-64ec35b22d85 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }, zpool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }, zpool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }, zpool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }, zpool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }, zpool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }, zpool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }, zpool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }, zpool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
4152024-05-08T03:32:24.918ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4162024-05-08T03:32:24.918ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 12193121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48070 req_id = 0e147a62-88ed-4be6-bda9-236f886c9e89 response_code = 200 uri = /omicron-physical-disks
4172024-05-08T03:32:24.951ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
4182024-05-08T03:32:24.951ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36908
4192024-05-08T03:32:24.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4202024-05-08T03:32:24.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4212024-05-08T03:32:25.003ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4222024-05-08T03:32:25.003ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
4232024-05-08T03:32:25.052ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4242024-05-08T03:32:25.052ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4252024-05-08T03:32:25.102ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4262024-05-08T03:32:25.102ZINFOSledAgent (ServiceManager): No ledger in /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:188
4272024-05-08T03:32:25.102ZINFOSledAgent (ServiceManager): No ledger in /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:188
4282024-05-08T03:32:25.102ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4292024-05-08T03:32:25.145ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4302024-05-08T03:32:25.145ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4312024-05-08T03:32:25.145ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4322024-05-08T03:32:25.175ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
4332024-05-08T03:32:25.197ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61 file = illumos-utils/src/zone.rs:315
4342024-05-08T03:32:25.201ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7 file = illumos-utils/src/zone.rs:315
4352024-05-08T03:32:25.205ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703 file = illumos-utils/src/zone.rs:315
4362024-05-08T03:32:25.217ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61 file = illumos-utils/src/zone.rs:354
4372024-05-08T03:32:25.222ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7 file = illumos-utils/src/zone.rs:354
4382024-05-08T03:32:25.227ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703 file = illumos-utils/src/zone.rs:354
4392024-05-08T03:32:26.177ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/411336b5-dbaf-48b1-93b3-a95d20f0f732): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
4402024-05-08T03:32:26.714ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4412024-05-08T03:32:26.714ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
4422024-05-08T03:32:26.714ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4432024-05-08T03:32:27.200ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4442024-05-08T03:32:27.680ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4452024-05-08T03:32:27.680ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
4462024-05-08T03:32:27.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 60.856651ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4472024-05-08T03:32:27.687ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
4482024-05-08T03:32:27.687ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4492024-05-08T03:32:27.691ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4502024-05-08T03:32:27.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 116.694476ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4512024-05-08T03:32:27.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 40.982059ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4522024-05-08T03:32:27.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61") failed: Property not found. retry in 56.874572ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4532024-05-08T03:32:27.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 75.468512ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4542024-05-08T03:32:27.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61") failed: Property not found. retry in 135.233147ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4552024-05-08T03:32:27.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 274.934785ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4562024-05-08T03:32:28.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 171.198627ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4572024-05-08T03:32:28.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61") failed: Property not found. retry in 135.241013ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4582024-05-08T03:32:28.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 552.283476ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4592024-05-08T03:32:28.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61") failed: Property not found. retry in 403.450393ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4602024-05-08T03:32:28.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 204.1125ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4612024-05-08T03:32:28.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 484.697024ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4622024-05-08T03:32:28.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61") failed: Property not found. retry in 1.179763166s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4632024-05-08T03:32:28.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 490.325978ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4642024-05-08T03:32:29.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 1.089869843s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4652024-05-08T03:32:29.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 995.485073ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4662024-05-08T03:32:29.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61") failed: Property not found. retry in 1.015291816s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4672024-05-08T03:32:30.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 535.708217ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4682024-05-08T03:32:30.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 760.003307ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4692024-05-08T03:32:30.372ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:9fb2::2 exists in switch zone file = sled-agent/src/services.rs:2320
4702024-05-08T03:32:30.372ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:626 zone = oxz_switch
4712024-05-08T03:32:30.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 1.497719789s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4722024-05-08T03:32:30.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61") failed: Property not found. retry in 1.220352103s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4732024-05-08T03:32:30.918ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fe66:5864 file = sled-agent/src/services.rs:2327
4742024-05-08T03:32:30.923ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2388
4752024-05-08T03:32:31.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 624.987175ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4762024-05-08T03:32:31.049ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2573
4772024-05-08T03:32:31.298ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2737
4782024-05-08T03:32:31.490ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2452
4792024-05-08T03:32:31.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 795.478374ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4802024-05-08T03:32:31.808ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2487
4812024-05-08T03:32:32.018ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2787
4822024-05-08T03:32:32.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61") failed: Property not found. retry in 922.731924ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4832024-05-08T03:32:32.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703") failed: Property not found. retry in 959.781397ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4842024-05-08T03:32:32.200ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2814
4852024-05-08T03:32:32.426ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2681
4862024-05-08T03:32:32.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7") failed: Property not found. retry in 869.286182ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4872024-05-08T03:32:32.630ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2481
4882024-05-08T03:32:32.641ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3873 new = [fd00:1122:3344:101::2, ::1] old = [::1]
4892024-05-08T03:32:32.641ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3889
4902024-05-08T03:32:32.641ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:609 zone = oxz_switch
4912024-05-08T03:32:33.207ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3897
4922024-05-08T03:32:33.213ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3953
4932024-05-08T03:32:33.347ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4015
4942024-05-08T03:32:33.364ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_dd2b8493-b48e-4834-ad8e-a53d21815a61
4952024-05-08T03:32:33.364ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_cc505437-f47e-4e85-84e3-c10ec252eef7
4962024-05-08T03:32:33.364ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_internal_dns_9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703
4972024-05-08T03:32:33.364ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
4982024-05-08T03:32:33.364ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
4992024-05-08T03:32:33.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8413682 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:36908 req_id = 0e5f39fe-221e-4371-a8ae-224bf9ea0bdd response_code = 204 uri = /omicron-zones
5002024-05-08T03:32:33.367ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:547
5012024-05-08T03:32:33.424ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 110.134897ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:567
5022024-05-08T03:32:33.446ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3996 rack_subnet = fd00:1122:3344::
5032024-05-08T03:32:33.469ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4043
5042024-05-08T03:32:33.556ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:547
5052024-05-08T03:32:33.567ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4070
5062024-05-08T03:32:33.619ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:547
5072024-05-08T03:32:33.684ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:583
5082024-05-08T03:32:33.684ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5092024-05-08T03:32:33.687ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:364
5102024-05-08T03:32:33.748ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5112024-05-08T03:32:33.750ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:412
5122024-05-08T03:32:33.750ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5132024-05-08T03:32:33.750ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5142024-05-08T03:32:33.806ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
5152024-05-08T03:32:33.807ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38630
5162024-05-08T03:32:33.807ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5172024-05-08T03:32:33.807ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5182024-05-08T03:32:33.812ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:589
5192024-05-08T03:32:33.812ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:627
5202024-05-08T03:32:33.813ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:630 uptime = 2
5212024-05-08T03:32:33.813ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:434
5222024-05-08T03:32:33.828ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5232024-05-08T03:32:33.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
5242024-05-08T03:32:33.846ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5252024-05-08T03:32:33.846ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5262024-05-08T03:32:33.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5272024-05-08T03:32:33.863ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
5282024-05-08T03:32:33.863ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
5292024-05-08T03:32:33.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5302024-05-08T03:32:33.877ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5312024-05-08T03:32:33.909ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
5322024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
5332024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
5342024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
5352024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
5362024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
5372024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
5382024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
5392024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
5402024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
5412024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
5422024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
5432024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
5442024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
5452024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
5462024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
5472024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
5482024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
5492024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
5502024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
5512024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
5522024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
5532024-05-08T03:32:33.910ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
5542024-05-08T03:32:33.919ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5552024-05-08T03:32:33.920ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5562024-05-08T03:32:33.920ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5572024-05-08T03:32:33.965ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 254, 79]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
5582024-05-08T03:32:33.965ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = ntp
5592024-05-08T03:32:33.966ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
5602024-05-08T03:32:33.966ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
5612024-05-08T03:32:33.966ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
5622024-05-08T03:32:33.968ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5632024-05-08T03:32:33.991ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed file = illumos-utils/src/zone.rs:315
5642024-05-08T03:32:34.010ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed file = illumos-utils/src/zone.rs:354
5652024-05-08T03:32:34.610ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5662024-05-08T03:32:34.610ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5672024-05-08T03:32:34.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 25.64676ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5682024-05-08T03:32:34.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 95.17991ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5692024-05-08T03:32:34.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 139.26904ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5702024-05-08T03:32:35.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 301.398524ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5712024-05-08T03:32:35.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 441.194036ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5722024-05-08T03:32:35.796ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 810.900031ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5732024-05-08T03:32:36.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 1.176822756s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5742024-05-08T03:32:37.090ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45359
5752024-05-08T03:32:37.091ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 75 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45359 req_id = 2a15ce44-33a1-4a5f-85f9-2903497ec2ac response_code = 200 uri = /baseboard
5762024-05-08T03:32:37.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 1.024015016s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5772024-05-08T03:32:38.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed") failed: Property not found. retry in 994.549401ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5782024-05-08T03:32:39.852ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed
5792024-05-08T03:32:39.852ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5802024-05-08T03:32:39.852ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5812024-05-08T03:32:39.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 6045937 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38630 req_id = 6e88f0bb-0cc7-40be-aa1c-bf67b259fb30 response_code = 204 uri = /omicron-zones
5822024-05-08T03:32:39.855ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:624
5832024-05-08T03:32:39.884ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:604
5842024-05-08T03:32:39.884ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60455
5852024-05-08T03:32:39.894ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
5862024-05-08T03:32:39.894ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 9668 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60455 req_id = 1d2884c7-ff3d-4e3e-bf37-41a65479c7be response_code = 500 uri = /timesync
5872024-05-08T03:32:39.897ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:654
5882024-05-08T03:32:40.060ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:604
5892024-05-08T03:32:40.060ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33421
5902024-05-08T03:32:44.069ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
5912024-05-08T03:32:44.069ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 4008860 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33421 req_id = 53a8cfb9-a554-4453-8ae3-9ee72eebda6d response_code = 500 uri = /timesync
5922024-05-08T03:32:44.072ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:654
5932024-05-08T03:32:44.281ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:604
5942024-05-08T03:32:44.281ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59919
5952024-05-08T03:32:44.290ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
5962024-05-08T03:32:44.291ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 8903 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59919 req_id = e9494a15-784d-4545-b6ec-4cd9bd2403ab response_code = 500 uri = /timesync
5972024-05-08T03:32:44.294ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:654
5982024-05-08T03:32:44.528ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:604
5992024-05-08T03:32:44.528ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50012
6002024-05-08T03:32:44.535ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_6d4f2f0d-0048-43a0-b797-4120ab6bb8ed': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3603
6012024-05-08T03:32:44.535ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 7682 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50012 req_id = 2bf2f528-ee4b-4315-a692-ca38476c29ad response_code = 500 uri = /timesync
6022024-05-08T03:32:44.537ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:654
6032024-05-08T03:32:44.761ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:604
6042024-05-08T03:32:44.761ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36786
6052024-05-08T03:32:48.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 4009056 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36786 req_id = 781b0ab8-4226-4ec0-908c-ec3a30e44e8d response_code = 200 uri = /timesync
6062024-05-08T03:32:48.773ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1715139163.6657124, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:632
6072024-05-08T03:32:48.773ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:654
6082024-05-08T03:32:49.046ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:604
6092024-05-08T03:32:49.046ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35500
6102024-05-08T03:32:49.055ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8883 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35500 req_id = 31e09e82-0745-4f71-b712-dcc9b2067932 response_code = 200 uri = /timesync
6112024-05-08T03:32:49.059ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1715139163.6657124, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:632
6122024-05-08T03:32:49.059ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:654
6132024-05-08T03:32:49.316ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:604
6142024-05-08T03:32:49.316ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46208
6152024-05-08T03:32:49.324ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:46208 req_id = fc2975b5-eccb-441b-b86c-23c2bdf1589c response_code = 200 uri = /timesync
6162024-05-08T03:32:49.327ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1715139163.6657124, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:632
6172024-05-08T03:32:49.327ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:654
6182024-05-08T03:32:49.010ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:604
6192024-05-08T03:32:49.010ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41588
6202024-05-08T03:32:49.014ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
6212024-05-08T03:32:49.031ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 20867 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41588 req_id = 467221b2-203b-4e8b-910b-e2549133ff20 response_code = 200 uri = /timesync
6222024-05-08T03:32:49.033ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2395651603, ip_addr: 142.202.190.19, stratum: 3, ref_time: 1715139168.8485675, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:632
6232024-05-08T03:32:49.033ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1192
6242024-05-08T03:32:49.062ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
6252024-05-08T03:32:49.062ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59595
6262024-05-08T03:32:49.063ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6272024-05-08T03:32:49.063ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6282024-05-08T03:32:49.084ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6292024-05-08T03:32:49.084ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
6302024-05-08T03:32:49.103ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6312024-05-08T03:32:49.103ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6322024-05-08T03:32:49.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6332024-05-08T03:32:49.120ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6342024-05-08T03:32:49.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6352024-05-08T03:32:49.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6362024-05-08T03:32:49.226ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6372024-05-08T03:32:49.226ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6382024-05-08T03:32:49.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6392024-05-08T03:32:49.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6402024-05-08T03:32:49.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6412024-05-08T03:32:49.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6422024-05-08T03:32:49.380ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6432024-05-08T03:32:49.380ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
6442024-05-08T03:32:49.380ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
6452024-05-08T03:32:49.380ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6462024-05-08T03:32:49.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6472024-05-08T03:32:49.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6482024-05-08T03:32:49.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6492024-05-08T03:32:49.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6502024-05-08T03:32:49.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6512024-05-08T03:32:49.554ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25 file = illumos-utils/src/zone.rs:315
6522024-05-08T03:32:49.558ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d file = illumos-utils/src/zone.rs:315
6532024-05-08T03:32:49.562ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120 file = illumos-utils/src/zone.rs:315
6542024-05-08T03:32:49.566ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709 file = illumos-utils/src/zone.rs:315
6552024-05-08T03:32:49.571ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5 file = illumos-utils/src/zone.rs:315
6562024-05-08T03:32:49.575ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25 file = illumos-utils/src/zone.rs:354
6572024-05-08T03:32:49.579ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d file = illumos-utils/src/zone.rs:354
6582024-05-08T03:32:49.584ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120 file = illumos-utils/src/zone.rs:354
6592024-05-08T03:32:49.588ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709 file = illumos-utils/src/zone.rs:354
6602024-05-08T03:32:49.593ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5 file = illumos-utils/src/zone.rs:354
6612024-05-08T03:32:51.439ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 321.822666ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
6622024-05-08T03:32:52.851ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::3"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::3"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6632024-05-08T03:32:52.851ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::4"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::4"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6642024-05-08T03:32:52.851ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
6652024-05-08T03:32:52.854ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
6662024-05-08T03:32:52.857ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
6672024-05-08T03:32:52.896ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::6"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6682024-05-08T03:32:52.896ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
6692024-05-08T03:32:52.938ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::5"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6702024-05-08T03:32:52.938ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
6712024-05-08T03:32:52.955ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::7"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6722024-05-08T03:32:52.955ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
6732024-05-08T03:32:53.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 58.964782ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
6742024-05-08T03:32:53.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 31.726592ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
6752024-05-08T03:32:53.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 59.69256ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
6762024-05-08T03:32:53.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 114.174639ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
6772024-05-08T03:32:53.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 148.643456ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
6782024-05-08T03:32:53.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 60.637878ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
6792024-05-08T03:32:53.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 67.39889ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
6802024-05-08T03:32:53.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 59.036758ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
6812024-05-08T03:32:54.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 216.831802ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
6822024-05-08T03:32:54.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 62.479001ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
6832024-05-08T03:32:54.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 178.433108ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
6842024-05-08T03:32:54.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 51.727317ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
6852024-05-08T03:32:54.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 192.595752ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
6862024-05-08T03:32:55.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 324.152317ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
6872024-05-08T03:32:55.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 256.260677ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
6882024-05-08T03:32:55.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 366.776872ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
6892024-05-08T03:32:55.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 115.545414ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
6902024-05-08T03:32:55.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 392.330261ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
6912024-05-08T03:32:55.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 565.686086ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
6922024-05-08T03:32:55.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 304.141871ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
6932024-05-08T03:32:55.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 581.581852ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
6942024-05-08T03:32:55.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 454.936774ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
6952024-05-08T03:32:55.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 409.317139ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
6962024-05-08T03:32:55.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 876.753889ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
6972024-05-08T03:32:56.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 662.881779ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
6982024-05-08T03:32:56.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 990.206685ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
6992024-05-08T03:32:56.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 601.327205ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
7002024-05-08T03:32:56.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 949.078475ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7012024-05-08T03:32:56.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 817.177909ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
7022024-05-08T03:32:56.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 1.481391195s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
7032024-05-08T03:32:56.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 1.133304032s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
7042024-05-08T03:32:57.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 1.093465529s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
7052024-05-08T03:32:57.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 712.256301ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7062024-05-08T03:32:57.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 512.96652ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
7072024-05-08T03:32:57.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 1.430567558s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7082024-05-08T03:32:58.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 1.403106239s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
7092024-05-08T03:32:58.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 607.048372ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
7102024-05-08T03:32:58.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 626.387275ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
7112024-05-08T03:32:58.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 1.247213134s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
7122024-05-08T03:32:58.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 576.247161ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
7132024-05-08T03:32:58.906ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 1.363314959s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
7142024-05-08T03:32:59.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 928.944017ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
7152024-05-08T03:32:59.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 980.725079ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7162024-05-08T03:32:59.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 1.139330304s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
7172024-05-08T03:32:59.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 940.114116ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
7182024-05-08T03:33:00.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5") failed: Property not found. retry in 1.498957952s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
7192024-05-08T03:33:00.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 954.914015ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
7202024-05-08T03:33:00.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 1.259272197s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7212024-05-08T03:33:00.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25") failed: Property not found. retry in 1.222144485s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
7222024-05-08T03:33:00.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d") failed: Property not found. retry in 1.285319631s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
7232024-05-08T03:33:01.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120") failed: Property not found. retry in 916.320822ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
7242024-05-08T03:33:01.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 1.260922985s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7252024-05-08T03:33:02.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 897.540795ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7262024-05-08T03:33:03.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 755.785398ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7272024-05-08T03:33:04.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 1.343746949s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7282024-05-08T03:33:06.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 1.495535641s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7292024-05-08T03:33:06.810ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65129
7302024-05-08T03:33:06.810ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65129 req_id = 1be8e7ed-b1c5-4a8f-9372-c08d59e6cad6 response_code = 200 uri = /baseboard
7312024-05-08T03:33:07.265ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 348.989711ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
7322024-05-08T03:33:07.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 1.135795916s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7332024-05-08T03:33:08.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 1.267229279s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7342024-05-08T03:33:09.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 941.289995ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7352024-05-08T03:33:10.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709") failed: Property not found. retry in 1.155043812s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7362024-05-08T03:33:12.088ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_8a2bf75b-8c7f-4d8d-9ede-6fc064124d25
7372024-05-08T03:33:12.088ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_45e11811-5e91-4520-b80a-7f642e885b6d
7382024-05-08T03:33:12.088ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_14c4878d-c5dd-4fc2-8846-aaf3152f4120
7392024-05-08T03:33:12.088ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_f44a281c-ca8a-47a1-9feb-a5b347543709
7402024-05-08T03:33:12.088ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_cockroachdb_ae06d4ce-795c-4d42-b79d-9db3f61d73a5
7412024-05-08T03:33:12.088ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7422024-05-08T03:33:12.089ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7432024-05-08T03:33:12.090ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 22558010 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59595 req_id = c9dea3d6-aac0-49e5-b1ea-8e6976f7ebb6 response_code = 204 uri = /omicron-zones
7442024-05-08T03:33:12.123ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54375
7452024-05-08T03:33:12.152ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::5]:32221 file = sled-agent/src/services.rs:3437
7462024-05-08T03:33:12.334ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3455
7472024-05-08T03:33:16.226ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3478
7482024-05-08T03:33:16.226ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 4103334 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:54375 req_id = 4ef6808f-d23a-472e-93e1-d050285370f1 response_code = 204 uri = /cockroachdb
7492024-05-08T03:33:16.260ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
7502024-05-08T03:33:16.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64762
7512024-05-08T03:33:16.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7522024-05-08T03:33:16.261ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7532024-05-08T03:33:16.284ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7542024-05-08T03:33:16.284ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
7552024-05-08T03:33:16.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7562024-05-08T03:33:16.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7572024-05-08T03:33:16.327ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7582024-05-08T03:33:16.327ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7592024-05-08T03:33:16.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7602024-05-08T03:33:16.347ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7612024-05-08T03:33:16.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7622024-05-08T03:33:16.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7632024-05-08T03:33:16.387ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7642024-05-08T03:33:16.387ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7652024-05-08T03:33:16.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7662024-05-08T03:33:16.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7672024-05-08T03:33:16.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7682024-05-08T03:33:16.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
7692024-05-08T03:33:16.507ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7702024-05-08T03:33:16.507ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
7712024-05-08T03:33:16.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
7722024-05-08T03:33:16.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
7732024-05-08T03:33:16.669ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7742024-05-08T03:33:16.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
7752024-05-08T03:33:16.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7762024-05-08T03:33:16.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
7772024-05-08T03:33:16.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7782024-05-08T03:33:16.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
7792024-05-08T03:33:16.908ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7802024-05-08T03:33:16.908ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
7812024-05-08T03:33:16.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7822024-05-08T03:33:16.982ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
7832024-05-08T03:33:17.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7842024-05-08T03:33:17.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
7852024-05-08T03:33:17.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7862024-05-08T03:33:17.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
7872024-05-08T03:33:17.180ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7882024-05-08T03:33:17.180ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
7892024-05-08T03:33:17.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
7902024-05-08T03:33:17.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
7912024-05-08T03:33:17.320ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7922024-05-08T03:33:17.320ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
7932024-05-08T03:33:17.321ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
7942024-05-08T03:33:17.321ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7952024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
7962024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
7972024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
7982024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
7992024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
8002024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
8012024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
8022024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
8032024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
8042024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
8052024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
8062024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
8072024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
8082024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
8092024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
8102024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
8112024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
8122024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
8132024-05-08T03:33:17.433ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
8142024-05-08T03:33:17.433ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 1172386 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 0ef94c10-fbf1-42b4-bdfa-d80c3d0f10b0 response_code = 500 uri = /omicron-zones
8152024-05-08T03:33:17.433ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0ef94c10-fbf1-42b4-bdfa-d80c3d0f10b0", "content-length": "124", "date": "Wed, 08 May 2024 03:33:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0ef94c10-fbf1-42b4-bdfa-d80c3d0f10b0" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 124.107936ms sled_address = [fd00:1122:3344:101::1]:12345
8162024-05-08T03:33:17.559ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
8172024-05-08T03:33:17.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8182024-05-08T03:33:17.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8192024-05-08T03:33:17.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8202024-05-08T03:33:17.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
8212024-05-08T03:33:17.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8222024-05-08T03:33:17.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8232024-05-08T03:33:17.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8242024-05-08T03:33:17.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8252024-05-08T03:33:17.650ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8262024-05-08T03:33:17.650ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8272024-05-08T03:33:17.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8282024-05-08T03:33:17.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8292024-05-08T03:33:17.691ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8302024-05-08T03:33:17.691ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8312024-05-08T03:33:17.711ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8322024-05-08T03:33:17.711ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8332024-05-08T03:33:17.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8342024-05-08T03:33:17.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8352024-05-08T03:33:17.753ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8362024-05-08T03:33:17.753ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8372024-05-08T03:33:17.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
8382024-05-08T03:33:17.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8392024-05-08T03:33:17.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8402024-05-08T03:33:17.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
8412024-05-08T03:33:17.815ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8422024-05-08T03:33:17.815ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
8432024-05-08T03:33:17.836ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8442024-05-08T03:33:17.836ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
8452024-05-08T03:33:17.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8462024-05-08T03:33:17.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
8472024-05-08T03:33:17.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8482024-05-08T03:33:17.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
8492024-05-08T03:33:17.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8502024-05-08T03:33:17.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
8512024-05-08T03:33:17.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8522024-05-08T03:33:17.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
8532024-05-08T03:33:17.939ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8542024-05-08T03:33:17.939ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
8552024-05-08T03:33:17.959ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8562024-05-08T03:33:17.959ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
8572024-05-08T03:33:17.980ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8582024-05-08T03:33:17.980ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
8592024-05-08T03:33:17.980ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
8602024-05-08T03:33:17.980ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8612024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
8622024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
8632024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
8642024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
8652024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
8662024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
8672024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
8682024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
8692024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
8702024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
8712024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
8722024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
8732024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
8742024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
8752024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
8762024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
8772024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
8782024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
8792024-05-08T03:33:18.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
8802024-05-08T03:33:18.078ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 519008 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 6bee72c8-1242-4c47-95c1-c845c726bf43 response_code = 500 uri = /omicron-zones
8812024-05-08T03:33:18.078ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6bee72c8-1242-4c47-95c1-c845c726bf43", "content-length": "124", "date": "Wed, 08 May 2024 03:33:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6bee72c8-1242-4c47-95c1-c845c726bf43" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 164.722531ms sled_address = [fd00:1122:3344:101::1]:12345
8822024-05-08T03:33:18.244ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
8832024-05-08T03:33:18.244ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8842024-05-08T03:33:18.244ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8852024-05-08T03:33:18.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8862024-05-08T03:33:18.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
8872024-05-08T03:33:18.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8882024-05-08T03:33:18.291ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8892024-05-08T03:33:18.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8902024-05-08T03:33:18.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8912024-05-08T03:33:18.333ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8922024-05-08T03:33:18.333ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8932024-05-08T03:33:18.355ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8942024-05-08T03:33:18.355ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8952024-05-08T03:33:18.374ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8962024-05-08T03:33:18.374ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8972024-05-08T03:33:18.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8982024-05-08T03:33:18.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8992024-05-08T03:33:18.416ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9002024-05-08T03:33:18.416ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9012024-05-08T03:33:18.438ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9022024-05-08T03:33:18.438ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9032024-05-08T03:33:18.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
9042024-05-08T03:33:18.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9052024-05-08T03:33:18.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9062024-05-08T03:33:18.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
9072024-05-08T03:33:18.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9082024-05-08T03:33:18.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
9092024-05-08T03:33:18.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9102024-05-08T03:33:18.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
9112024-05-08T03:33:18.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9122024-05-08T03:33:18.548ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
9132024-05-08T03:33:18.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9142024-05-08T03:33:18.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
9152024-05-08T03:33:18.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9162024-05-08T03:33:18.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
9172024-05-08T03:33:18.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9182024-05-08T03:33:18.615ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
9192024-05-08T03:33:18.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9202024-05-08T03:33:18.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
9212024-05-08T03:33:18.657ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9222024-05-08T03:33:18.657ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
9232024-05-08T03:33:18.677ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9242024-05-08T03:33:18.678ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
9252024-05-08T03:33:18.678ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
9262024-05-08T03:33:18.678ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9272024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
9282024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
9292024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
9302024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
9312024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
9322024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
9332024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
9342024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
9352024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
9362024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
9372024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
9382024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
9392024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
9402024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
9412024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
9422024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
9432024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
9442024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
9452024-05-08T03:33:18.782ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
9462024-05-08T03:33:18.782ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 538035 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = c67478f7-f963-46e3-a888-79b2926dee5d response_code = 500 uri = /omicron-zones
9472024-05-08T03:33:18.782ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c67478f7-f963-46e3-a888-79b2926dee5d", "content-length": "124", "date": "Wed, 08 May 2024 03:33:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c67478f7-f963-46e3-a888-79b2926dee5d" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 76.98722ms sled_address = [fd00:1122:3344:101::1]:12345
9482024-05-08T03:33:18.860ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
9492024-05-08T03:33:18.861ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9502024-05-08T03:33:18.861ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9512024-05-08T03:33:18.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9522024-05-08T03:33:18.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
9532024-05-08T03:33:18.908ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9542024-05-08T03:33:18.908ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9552024-05-08T03:33:18.931ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9562024-05-08T03:33:18.931ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9572024-05-08T03:33:18.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9582024-05-08T03:33:18.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9592024-05-08T03:33:18.973ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9602024-05-08T03:33:18.973ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9612024-05-08T03:33:18.993ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9622024-05-08T03:33:18.993ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9632024-05-08T03:33:19.014ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9642024-05-08T03:33:19.014ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9652024-05-08T03:33:19.034ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9662024-05-08T03:33:19.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9672024-05-08T03:33:19.053ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9682024-05-08T03:33:19.053ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9692024-05-08T03:33:19.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
9702024-05-08T03:33:19.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9712024-05-08T03:33:19.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9722024-05-08T03:33:19.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
9732024-05-08T03:33:19.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9742024-05-08T03:33:19.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
9752024-05-08T03:33:19.138ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9762024-05-08T03:33:19.138ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
9772024-05-08T03:33:19.158ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9782024-05-08T03:33:19.158ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
9792024-05-08T03:33:19.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9802024-05-08T03:33:19.177ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
9812024-05-08T03:33:19.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9822024-05-08T03:33:19.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
9832024-05-08T03:33:19.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9842024-05-08T03:33:19.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
9852024-05-08T03:33:19.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9862024-05-08T03:33:19.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
9872024-05-08T03:33:19.254ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9882024-05-08T03:33:19.254ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
9892024-05-08T03:33:19.273ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9902024-05-08T03:33:19.273ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
9912024-05-08T03:33:19.273ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
9922024-05-08T03:33:19.273ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9932024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
9942024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
9952024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
9962024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
9972024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
9982024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
9992024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
10002024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
10012024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
10022024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
10032024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
10042024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
10052024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
10062024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
10072024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
10082024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
10092024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
10102024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
10112024-05-08T03:33:19.373ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
10122024-05-08T03:33:19.373ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 512838 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = b52c94f3-cd06-4e76-8767-d296c683f35f response_code = 500 uri = /omicron-zones
10132024-05-08T03:33:19.374ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b52c94f3-cd06-4e76-8767-d296c683f35f", "content-length": "124", "date": "Wed, 08 May 2024 03:33:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b52c94f3-cd06-4e76-8767-d296c683f35f" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 235.807319ms sled_address = [fd00:1122:3344:101::1]:12345
10142024-05-08T03:33:19.611ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
10152024-05-08T03:33:19.611ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10162024-05-08T03:33:19.611ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10172024-05-08T03:33:19.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10182024-05-08T03:33:19.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
10192024-05-08T03:33:19.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10202024-05-08T03:33:19.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10212024-05-08T03:33:19.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10222024-05-08T03:33:19.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10232024-05-08T03:33:19.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10242024-05-08T03:33:19.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10252024-05-08T03:33:19.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10262024-05-08T03:33:19.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10272024-05-08T03:33:19.731ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10282024-05-08T03:33:19.731ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10292024-05-08T03:33:19.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10302024-05-08T03:33:19.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10312024-05-08T03:33:19.768ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10322024-05-08T03:33:19.768ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10332024-05-08T03:33:19.790ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10342024-05-08T03:33:19.790ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10352024-05-08T03:33:19.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
10362024-05-08T03:33:19.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10372024-05-08T03:33:19.828ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10382024-05-08T03:33:19.828ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
10392024-05-08T03:33:19.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10402024-05-08T03:33:19.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
10412024-05-08T03:33:19.869ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10422024-05-08T03:33:19.869ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
10432024-05-08T03:33:19.888ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10442024-05-08T03:33:19.888ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
10452024-05-08T03:33:19.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10462024-05-08T03:33:19.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
10472024-05-08T03:33:19.930ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10482024-05-08T03:33:19.930ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
10492024-05-08T03:33:19.951ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10502024-05-08T03:33:19.951ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
10512024-05-08T03:33:19.971ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10522024-05-08T03:33:19.971ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
10532024-05-08T03:33:19.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10542024-05-08T03:33:19.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
10552024-05-08T03:33:20.011ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10562024-05-08T03:33:20.011ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
10572024-05-08T03:33:20.011ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
10582024-05-08T03:33:20.011ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10592024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
10602024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
10612024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
10622024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
10632024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
10642024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
10652024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
10662024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
10672024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
10682024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
10692024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
10702024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
10712024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
10722024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
10732024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
10742024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
10752024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
10762024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
10772024-05-08T03:33:20.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
10782024-05-08T03:33:20.107ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 496282 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = af531150-d092-45ca-b74a-4d55561645ad response_code = 500 uri = /omicron-zones
10792024-05-08T03:33:20.107ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "af531150-d092-45ca-b74a-4d55561645ad", "content-length": "124", "date": "Wed, 08 May 2024 03:33:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "af531150-d092-45ca-b74a-4d55561645ad" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 204.72719ms sled_address = [fd00:1122:3344:101::1]:12345
10802024-05-08T03:33:20.313ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
10812024-05-08T03:33:20.314ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10822024-05-08T03:33:20.314ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10832024-05-08T03:33:20.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10842024-05-08T03:33:20.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
10852024-05-08T03:33:20.361ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10862024-05-08T03:33:20.361ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10872024-05-08T03:33:20.380ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10882024-05-08T03:33:20.380ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10892024-05-08T03:33:20.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10902024-05-08T03:33:20.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10912024-05-08T03:33:20.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10922024-05-08T03:33:20.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10932024-05-08T03:33:20.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10942024-05-08T03:33:20.437ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10952024-05-08T03:33:20.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10962024-05-08T03:33:20.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10972024-05-08T03:33:20.480ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10982024-05-08T03:33:20.480ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10992024-05-08T03:33:20.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11002024-05-08T03:33:20.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11012024-05-08T03:33:20.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
11022024-05-08T03:33:20.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11032024-05-08T03:33:20.540ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11042024-05-08T03:33:20.540ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
11052024-05-08T03:33:20.560ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11062024-05-08T03:33:20.560ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
11072024-05-08T03:33:20.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11082024-05-08T03:33:20.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
11092024-05-08T03:33:20.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11102024-05-08T03:33:20.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
11112024-05-08T03:33:20.618ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11122024-05-08T03:33:20.618ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
11132024-05-08T03:33:20.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11142024-05-08T03:33:20.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
11152024-05-08T03:33:20.655ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11162024-05-08T03:33:20.655ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
11172024-05-08T03:33:20.675ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11182024-05-08T03:33:20.675ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
11192024-05-08T03:33:20.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11202024-05-08T03:33:20.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
11212024-05-08T03:33:20.713ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11222024-05-08T03:33:20.713ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
11232024-05-08T03:33:20.714ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
11242024-05-08T03:33:20.714ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11252024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
11262024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
11272024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
11282024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
11292024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
11302024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
11312024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
11322024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
11332024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
11342024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
11352024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
11362024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
11372024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
11382024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
11392024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
11402024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
11412024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
11422024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
11432024-05-08T03:33:20.806ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
11442024-05-08T03:33:20.806ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 493021 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 242b5fad-1f87-4688-981b-82221cc80276 response_code = 500 uri = /omicron-zones
11452024-05-08T03:33:20.806ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "242b5fad-1f87-4688-981b-82221cc80276", "content-length": "124", "date": "Wed, 08 May 2024 03:33:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "242b5fad-1f87-4688-981b-82221cc80276" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 186.208505ms sled_address = [fd00:1122:3344:101::1]:12345
11462024-05-08T03:33:20.994ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
11472024-05-08T03:33:20.994ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11482024-05-08T03:33:20.994ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11492024-05-08T03:33:21.014ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11502024-05-08T03:33:21.014ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
11512024-05-08T03:33:21.034ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11522024-05-08T03:33:21.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11532024-05-08T03:33:21.052ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11542024-05-08T03:33:21.052ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11552024-05-08T03:33:21.071ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11562024-05-08T03:33:21.071ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11572024-05-08T03:33:21.088ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11582024-05-08T03:33:21.089ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11592024-05-08T03:33:21.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11602024-05-08T03:33:21.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11612024-05-08T03:33:21.123ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11622024-05-08T03:33:21.123ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11632024-05-08T03:33:21.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11642024-05-08T03:33:21.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11652024-05-08T03:33:21.159ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11662024-05-08T03:33:21.159ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11672024-05-08T03:33:21.176ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
11682024-05-08T03:33:21.176ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11692024-05-08T03:33:21.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11702024-05-08T03:33:21.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
11712024-05-08T03:33:21.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11722024-05-08T03:33:21.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
11732024-05-08T03:33:21.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11742024-05-08T03:33:21.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
11752024-05-08T03:33:21.247ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11762024-05-08T03:33:21.247ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
11772024-05-08T03:33:21.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11782024-05-08T03:33:21.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
11792024-05-08T03:33:21.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11802024-05-08T03:33:21.283ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
11812024-05-08T03:33:21.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11822024-05-08T03:33:21.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
11832024-05-08T03:33:21.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11842024-05-08T03:33:21.320ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
11852024-05-08T03:33:21.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11862024-05-08T03:33:21.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
11872024-05-08T03:33:21.356ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11882024-05-08T03:33:21.356ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
11892024-05-08T03:33:21.356ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
11902024-05-08T03:33:21.356ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11912024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
11922024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
11932024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
11942024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
11952024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
11962024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
11972024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
11982024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
11992024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
12002024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
12012024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
12022024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
12032024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
12042024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
12052024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
12062024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
12072024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
12082024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
12092024-05-08T03:33:21.450ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
12102024-05-08T03:33:21.450ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 456553 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 7ef614c6-f824-4dc5-9371-d4b411775ad0 response_code = 500 uri = /omicron-zones
12112024-05-08T03:33:21.451ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7ef614c6-f824-4dc5-9371-d4b411775ad0", "content-length": "124", "date": "Wed, 08 May 2024 03:33:21 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7ef614c6-f824-4dc5-9371-d4b411775ad0" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 265.496056ms sled_address = [fd00:1122:3344:101::1]:12345
12122024-05-08T03:33:21.717ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
12132024-05-08T03:33:21.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12142024-05-08T03:33:21.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12152024-05-08T03:33:21.742ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12162024-05-08T03:33:21.742ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
12172024-05-08T03:33:21.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12182024-05-08T03:33:21.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12192024-05-08T03:33:21.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12202024-05-08T03:33:21.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12212024-05-08T03:33:21.804ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12222024-05-08T03:33:21.804ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12232024-05-08T03:33:21.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12242024-05-08T03:33:21.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12252024-05-08T03:33:21.846ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12262024-05-08T03:33:21.846ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12272024-05-08T03:33:21.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12282024-05-08T03:33:21.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12292024-05-08T03:33:21.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12302024-05-08T03:33:21.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12312024-05-08T03:33:21.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12322024-05-08T03:33:21.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12332024-05-08T03:33:21.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
12342024-05-08T03:33:21.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12352024-05-08T03:33:21.949ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12362024-05-08T03:33:21.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
12372024-05-08T03:33:21.969ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12382024-05-08T03:33:21.969ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
12392024-05-08T03:33:21.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12402024-05-08T03:33:21.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
12412024-05-08T03:33:22.010ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12422024-05-08T03:33:22.010ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
12432024-05-08T03:33:22.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12442024-05-08T03:33:22.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
12452024-05-08T03:33:22.053ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12462024-05-08T03:33:22.053ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
12472024-05-08T03:33:22.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12482024-05-08T03:33:22.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
12492024-05-08T03:33:22.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12502024-05-08T03:33:22.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
12512024-05-08T03:33:22.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12522024-05-08T03:33:22.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
12532024-05-08T03:33:22.135ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12542024-05-08T03:33:22.135ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
12552024-05-08T03:33:22.135ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
12562024-05-08T03:33:22.135ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12572024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
12582024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
12592024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
12602024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
12612024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
12622024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
12632024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
12642024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
12652024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
12662024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
12672024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
12682024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
12692024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
12702024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
12712024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
12722024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
12732024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
12742024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
12752024-05-08T03:33:22.231ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
12762024-05-08T03:33:22.231ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 513979 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 629cb140-6e9a-4d6f-a875-0d3476ba7544 response_code = 500 uri = /omicron-zones
12772024-05-08T03:33:22.231ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "629cb140-6e9a-4d6f-a875-0d3476ba7544", "content-length": "124", "date": "Wed, 08 May 2024 03:33:21 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "629cb140-6e9a-4d6f-a875-0d3476ba7544" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 338.433134ms sled_address = [fd00:1122:3344:101::1]:12345
12782024-05-08T03:33:22.571ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
12792024-05-08T03:33:22.571ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12802024-05-08T03:33:22.571ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12812024-05-08T03:33:22.593ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12822024-05-08T03:33:22.593ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
12832024-05-08T03:33:22.611ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12842024-05-08T03:33:22.611ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12852024-05-08T03:33:22.629ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12862024-05-08T03:33:22.629ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12872024-05-08T03:33:22.647ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12882024-05-08T03:33:22.647ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12892024-05-08T03:33:22.665ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 581.374779ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
12902024-05-08T03:33:22.666ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12912024-05-08T03:33:22.666ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12922024-05-08T03:33:22.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12932024-05-08T03:33:22.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12942024-05-08T03:33:22.701ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12952024-05-08T03:33:22.701ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12962024-05-08T03:33:22.719ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12972024-05-08T03:33:22.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12982024-05-08T03:33:22.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12992024-05-08T03:33:22.737ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13002024-05-08T03:33:22.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
13012024-05-08T03:33:22.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13022024-05-08T03:33:22.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13032024-05-08T03:33:22.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
13042024-05-08T03:33:22.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13052024-05-08T03:33:22.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
13062024-05-08T03:33:22.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13072024-05-08T03:33:22.811ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
13082024-05-08T03:33:22.831ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13092024-05-08T03:33:22.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
13102024-05-08T03:33:22.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13112024-05-08T03:33:22.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
13122024-05-08T03:33:22.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13132024-05-08T03:33:22.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
13142024-05-08T03:33:22.892ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13152024-05-08T03:33:22.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
13162024-05-08T03:33:22.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13172024-05-08T03:33:22.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
13182024-05-08T03:33:22.931ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13192024-05-08T03:33:22.931ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
13202024-05-08T03:33:22.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13212024-05-08T03:33:22.949ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
13222024-05-08T03:33:22.949ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
13232024-05-08T03:33:22.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13242024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
13252024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
13262024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
13272024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
13282024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
13292024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
13302024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
13312024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
13322024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
13332024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
13342024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
13352024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
13362024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
13372024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
13382024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
13392024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
13402024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
13412024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
13422024-05-08T03:33:23.043ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
13432024-05-08T03:33:23.043ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 471991 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = e0fdc4ef-bb40-4421-acea-f3b2ce7d5e5a response_code = 500 uri = /omicron-zones
13442024-05-08T03:33:23.043ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e0fdc4ef-bb40-4421-acea-f3b2ce7d5e5a", "content-length": "124", "date": "Wed, 08 May 2024 03:33:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e0fdc4ef-bb40-4421-acea-f3b2ce7d5e5a" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 460.769219ms sled_address = [fd00:1122:3344:101::1]:12345
13452024-05-08T03:33:23.505ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
13462024-05-08T03:33:23.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13472024-05-08T03:33:23.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13482024-05-08T03:33:23.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13492024-05-08T03:33:23.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
13502024-05-08T03:33:23.553ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13512024-05-08T03:33:23.553ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13522024-05-08T03:33:23.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13532024-05-08T03:33:23.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13542024-05-08T03:33:23.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13552024-05-08T03:33:23.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13562024-05-08T03:33:23.613ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13572024-05-08T03:33:23.613ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13582024-05-08T03:33:23.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13592024-05-08T03:33:23.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13602024-05-08T03:33:23.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13612024-05-08T03:33:23.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13622024-05-08T03:33:23.671ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13632024-05-08T03:33:23.671ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13642024-05-08T03:33:23.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13652024-05-08T03:33:23.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13662024-05-08T03:33:23.712ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
13672024-05-08T03:33:23.712ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13682024-05-08T03:33:23.733ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13692024-05-08T03:33:23.733ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
13702024-05-08T03:33:23.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13712024-05-08T03:33:23.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
13722024-05-08T03:33:23.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13732024-05-08T03:33:23.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
13742024-05-08T03:33:23.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13752024-05-08T03:33:23.797ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
13762024-05-08T03:33:23.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13772024-05-08T03:33:23.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
13782024-05-08T03:33:23.842ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13792024-05-08T03:33:23.842ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
13802024-05-08T03:33:23.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13812024-05-08T03:33:23.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
13822024-05-08T03:33:23.885ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13832024-05-08T03:33:23.885ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
13842024-05-08T03:33:23.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13852024-05-08T03:33:23.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
13862024-05-08T03:33:23.923ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13872024-05-08T03:33:23.923ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
13882024-05-08T03:33:23.923ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
13892024-05-08T03:33:23.923ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13902024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
13912024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
13922024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
13932024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
13942024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
13952024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
13962024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
13972024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
13982024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
13992024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
14002024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
14012024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
14022024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
14032024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
14042024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
14052024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
14062024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
14072024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
14082024-05-08T03:33:24.019ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
14092024-05-08T03:33:24.019ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 513939 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = df64001f-51c4-4358-a96b-ed96e37546ca response_code = 500 uri = /omicron-zones
14102024-05-08T03:33:24.019ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "df64001f-51c4-4358-a96b-ed96e37546ca", "content-length": "124", "date": "Wed, 08 May 2024 03:33:23 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "df64001f-51c4-4358-a96b-ed96e37546ca" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 390.768995ms sled_address = [fd00:1122:3344:101::1]:12345
14112024-05-08T03:33:24.412ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
14122024-05-08T03:33:24.412ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14132024-05-08T03:33:24.413ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14142024-05-08T03:33:24.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14152024-05-08T03:33:24.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
14162024-05-08T03:33:24.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14172024-05-08T03:33:24.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14182024-05-08T03:33:24.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14192024-05-08T03:33:24.475ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14202024-05-08T03:33:24.494ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14212024-05-08T03:33:24.494ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14222024-05-08T03:33:24.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14232024-05-08T03:33:24.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14242024-05-08T03:33:24.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14252024-05-08T03:33:24.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14262024-05-08T03:33:24.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14272024-05-08T03:33:24.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14282024-05-08T03:33:24.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14292024-05-08T03:33:24.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14302024-05-08T03:33:24.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14312024-05-08T03:33:24.595ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14322024-05-08T03:33:24.619ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
14332024-05-08T03:33:24.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14342024-05-08T03:33:24.647ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14352024-05-08T03:33:24.647ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
14362024-05-08T03:33:24.671ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14372024-05-08T03:33:24.671ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
14382024-05-08T03:33:24.696ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14392024-05-08T03:33:24.697ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
14402024-05-08T03:33:24.721ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14412024-05-08T03:33:24.721ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
14422024-05-08T03:33:24.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14432024-05-08T03:33:24.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
14442024-05-08T03:33:24.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14452024-05-08T03:33:24.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
14462024-05-08T03:33:24.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14472024-05-08T03:33:24.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
14482024-05-08T03:33:24.820ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14492024-05-08T03:33:24.820ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
14502024-05-08T03:33:24.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14512024-05-08T03:33:24.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
14522024-05-08T03:33:24.870ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14532024-05-08T03:33:24.870ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
14542024-05-08T03:33:24.870ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
14552024-05-08T03:33:24.870ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14562024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
14572024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
14582024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
14592024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
14602024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
14612024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
14622024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
14632024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
14642024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
14652024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
14662024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
14672024-05-08T03:33:24.981ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
14682024-05-08T03:33:24.982ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
14692024-05-08T03:33:24.982ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
14702024-05-08T03:33:24.982ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
14712024-05-08T03:33:24.982ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
14722024-05-08T03:33:24.982ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
14732024-05-08T03:33:24.982ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
14742024-05-08T03:33:24.982ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
14752024-05-08T03:33:24.982ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 569434 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = d8b98227-5fa2-4251-a356-c45aa57c6482 response_code = 500 uri = /omicron-zones
14762024-05-08T03:33:24.982ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d8b98227-5fa2-4251-a356-c45aa57c6482", "content-length": "124", "date": "Wed, 08 May 2024 03:33:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d8b98227-5fa2-4251-a356-c45aa57c6482" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 374.092592ms sled_address = [fd00:1122:3344:101::1]:12345
14772024-05-08T03:33:25.356ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
14782024-05-08T03:33:25.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14792024-05-08T03:33:25.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14802024-05-08T03:33:25.378ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14812024-05-08T03:33:25.379ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
14822024-05-08T03:33:25.399ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14832024-05-08T03:33:25.399ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14842024-05-08T03:33:25.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14852024-05-08T03:33:25.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14862024-05-08T03:33:25.437ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14872024-05-08T03:33:25.437ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14882024-05-08T03:33:25.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14892024-05-08T03:33:25.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14902024-05-08T03:33:25.474ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14912024-05-08T03:33:25.474ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14922024-05-08T03:33:25.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14932024-05-08T03:33:25.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14942024-05-08T03:33:25.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14952024-05-08T03:33:25.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14962024-05-08T03:33:25.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14972024-05-08T03:33:25.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14982024-05-08T03:33:25.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
14992024-05-08T03:33:25.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15002024-05-08T03:33:25.571ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15012024-05-08T03:33:25.571ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
15022024-05-08T03:33:25.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15032024-05-08T03:33:25.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
15042024-05-08T03:33:25.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15052024-05-08T03:33:25.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
15062024-05-08T03:33:25.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15072024-05-08T03:33:25.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
15082024-05-08T03:33:25.647ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15092024-05-08T03:33:25.647ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
15102024-05-08T03:33:25.666ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15112024-05-08T03:33:25.666ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
15122024-05-08T03:33:25.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15132024-05-08T03:33:25.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
15142024-05-08T03:33:25.702ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15152024-05-08T03:33:25.702ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
15162024-05-08T03:33:25.720ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15172024-05-08T03:33:25.720ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
15182024-05-08T03:33:25.738ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15192024-05-08T03:33:25.738ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
15202024-05-08T03:33:25.738ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
15212024-05-08T03:33:25.738ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15222024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
15232024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
15242024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
15252024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
15262024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
15272024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
15282024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
15292024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
15302024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
15312024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
15322024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
15332024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
15342024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
15352024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
15362024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
15372024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
15382024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
15392024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
15402024-05-08T03:33:25.832ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
15412024-05-08T03:33:25.832ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 475653 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 2e7f830a-cf50-42c0-9d7b-5cec7ab409e4 response_code = 500 uri = /omicron-zones
15422024-05-08T03:33:25.833ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2e7f830a-cf50-42c0-9d7b-5cec7ab409e4", "content-length": "124", "date": "Wed, 08 May 2024 03:33:25 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2e7f830a-cf50-42c0-9d7b-5cec7ab409e4" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 876.437488ms sled_address = [fd00:1122:3344:101::1]:12345
15432024-05-08T03:33:26.710ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
15442024-05-08T03:33:26.710ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15452024-05-08T03:33:26.710ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15462024-05-08T03:33:26.733ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15472024-05-08T03:33:26.733ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
15482024-05-08T03:33:26.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15492024-05-08T03:33:26.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15502024-05-08T03:33:26.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15512024-05-08T03:33:26.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15522024-05-08T03:33:26.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15532024-05-08T03:33:26.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15542024-05-08T03:33:26.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15552024-05-08T03:33:26.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15562024-05-08T03:33:26.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15572024-05-08T03:33:26.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15582024-05-08T03:33:26.851ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15592024-05-08T03:33:26.851ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15602024-05-08T03:33:26.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15612024-05-08T03:33:26.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15622024-05-08T03:33:26.891ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15632024-05-08T03:33:26.891ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15642024-05-08T03:33:26.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
15652024-05-08T03:33:26.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15662024-05-08T03:33:26.934ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15672024-05-08T03:33:26.934ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
15682024-05-08T03:33:26.954ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15692024-05-08T03:33:26.954ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
15702024-05-08T03:33:26.975ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15712024-05-08T03:33:26.976ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
15722024-05-08T03:33:26.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15732024-05-08T03:33:26.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
15742024-05-08T03:33:27.014ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15752024-05-08T03:33:27.014ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
15762024-05-08T03:33:27.033ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15772024-05-08T03:33:27.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
15782024-05-08T03:33:27.055ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15792024-05-08T03:33:27.055ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
15802024-05-08T03:33:27.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15812024-05-08T03:33:27.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
15822024-05-08T03:33:27.092ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15832024-05-08T03:33:27.092ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
15842024-05-08T03:33:27.111ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15852024-05-08T03:33:27.111ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
15862024-05-08T03:33:27.111ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
15872024-05-08T03:33:27.111ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15882024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
15892024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
15902024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
15912024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
15922024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
15932024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
15942024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
15952024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
15962024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
15972024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
15982024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
15992024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
16002024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
16012024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
16022024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
16032024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
16042024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
16052024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
16062024-05-08T03:33:27.203ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
16072024-05-08T03:33:27.203ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 493341 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 1494a8fb-a3a7-49b7-8bf9-79c81be99913 response_code = 500 uri = /omicron-zones
16082024-05-08T03:33:27.204ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1494a8fb-a3a7-49b7-8bf9-79c81be99913", "content-length": "124", "date": "Wed, 08 May 2024 03:33:26 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1494a8fb-a3a7-49b7-8bf9-79c81be99913" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 1.16875311s sled_address = [fd00:1122:3344:101::1]:12345
16092024-05-08T03:33:28.374ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
16102024-05-08T03:33:28.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16112024-05-08T03:33:28.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16122024-05-08T03:33:28.397ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16132024-05-08T03:33:28.397ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
16142024-05-08T03:33:28.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16152024-05-08T03:33:28.418ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16162024-05-08T03:33:28.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16172024-05-08T03:33:28.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16182024-05-08T03:33:28.453ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16192024-05-08T03:33:28.453ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16202024-05-08T03:33:28.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16212024-05-08T03:33:28.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16222024-05-08T03:33:28.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16232024-05-08T03:33:28.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16242024-05-08T03:33:28.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16252024-05-08T03:33:28.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16262024-05-08T03:33:28.524ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16272024-05-08T03:33:28.524ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16282024-05-08T03:33:28.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16292024-05-08T03:33:28.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16302024-05-08T03:33:28.561ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
16312024-05-08T03:33:28.561ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16322024-05-08T03:33:28.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16332024-05-08T03:33:28.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
16342024-05-08T03:33:28.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16352024-05-08T03:33:28.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
16362024-05-08T03:33:28.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16372024-05-08T03:33:28.615ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
16382024-05-08T03:33:28.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16392024-05-08T03:33:28.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
16402024-05-08T03:33:28.651ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16412024-05-08T03:33:28.651ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
16422024-05-08T03:33:28.668ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16432024-05-08T03:33:28.668ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
16442024-05-08T03:33:28.688ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16452024-05-08T03:33:28.688ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
16462024-05-08T03:33:28.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16472024-05-08T03:33:28.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
16482024-05-08T03:33:28.726ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16492024-05-08T03:33:28.726ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
16502024-05-08T03:33:28.746ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16512024-05-08T03:33:28.746ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
16522024-05-08T03:33:28.746ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
16532024-05-08T03:33:28.747ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16542024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
16552024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
16562024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
16572024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
16582024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
16592024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
16602024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
16612024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
16622024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
16632024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
16642024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
16652024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
16662024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
16672024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
16682024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
16692024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
16702024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
16712024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
16722024-05-08T03:33:28.841ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
16732024-05-08T03:33:28.842ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 466867 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 4a1e627f-85a6-45ee-af4f-488c2adc5ca6 response_code = 500 uri = /omicron-zones
16742024-05-08T03:33:28.842ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4a1e627f-85a6-45ee-af4f-488c2adc5ca6", "content-length": "124", "date": "Wed, 08 May 2024 03:33:28 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4a1e627f-85a6-45ee-af4f-488c2adc5ca6" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 1.069011227s sled_address = [fd00:1122:3344:101::1]:12345
16752024-05-08T03:33:29.912ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
16762024-05-08T03:33:29.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16772024-05-08T03:33:29.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16782024-05-08T03:33:29.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16792024-05-08T03:33:29.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
16802024-05-08T03:33:29.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16812024-05-08T03:33:29.957ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16822024-05-08T03:33:29.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16832024-05-08T03:33:29.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16842024-05-08T03:33:29.996ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16852024-05-08T03:33:29.996ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16862024-05-08T03:33:30.015ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16872024-05-08T03:33:30.015ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16882024-05-08T03:33:30.034ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16892024-05-08T03:33:30.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16902024-05-08T03:33:30.053ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16912024-05-08T03:33:30.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16922024-05-08T03:33:30.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16932024-05-08T03:33:30.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16942024-05-08T03:33:30.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16952024-05-08T03:33:30.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16962024-05-08T03:33:30.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
16972024-05-08T03:33:30.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16982024-05-08T03:33:30.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16992024-05-08T03:33:30.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
17002024-05-08T03:33:30.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17012024-05-08T03:33:30.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
17022024-05-08T03:33:30.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17032024-05-08T03:33:30.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
17042024-05-08T03:33:30.192ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17052024-05-08T03:33:30.192ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
17062024-05-08T03:33:30.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17072024-05-08T03:33:30.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
17082024-05-08T03:33:30.229ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17092024-05-08T03:33:30.229ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
17102024-05-08T03:33:30.248ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17112024-05-08T03:33:30.248ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
17122024-05-08T03:33:30.267ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17132024-05-08T03:33:30.267ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17142024-05-08T03:33:30.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17152024-05-08T03:33:30.285ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
17162024-05-08T03:33:30.304ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17172024-05-08T03:33:30.304ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
17182024-05-08T03:33:30.304ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
17192024-05-08T03:33:30.304ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17202024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
17212024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
17222024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
17232024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
17242024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
17252024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
17262024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
17272024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
17282024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
17292024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
17302024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
17312024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
17322024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
17332024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
17342024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
17352024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
17362024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
17372024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
17382024-05-08T03:33:30.397ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
17392024-05-08T03:33:30.397ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 485151 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = faa0beed-3116-4e43-af5c-e7f07c990390 response_code = 500 uri = /omicron-zones
17402024-05-08T03:33:30.397ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "faa0beed-3116-4e43-af5c-e7f07c990390", "content-length": "124", "date": "Wed, 08 May 2024 03:33:29 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "faa0beed-3116-4e43-af5c-e7f07c990390" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 1.543455881s sled_address = [fd00:1122:3344:101::1]:12345
17412024-05-08T03:33:31.942ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
17422024-05-08T03:33:31.943ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17432024-05-08T03:33:31.943ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17442024-05-08T03:33:31.966ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17452024-05-08T03:33:31.966ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
17462024-05-08T03:33:31.985ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17472024-05-08T03:33:31.985ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17482024-05-08T03:33:32.006ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17492024-05-08T03:33:32.006ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17502024-05-08T03:33:32.026ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17512024-05-08T03:33:32.026ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17522024-05-08T03:33:32.045ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17532024-05-08T03:33:32.045ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17542024-05-08T03:33:32.063ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17552024-05-08T03:33:32.063ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17562024-05-08T03:33:32.083ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17572024-05-08T03:33:32.083ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17582024-05-08T03:33:32.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17592024-05-08T03:33:32.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17602024-05-08T03:33:32.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17612024-05-08T03:33:32.120ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17622024-05-08T03:33:32.138ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
17632024-05-08T03:33:32.138ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17642024-05-08T03:33:32.157ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17652024-05-08T03:33:32.157ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
17662024-05-08T03:33:32.173ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17672024-05-08T03:33:32.173ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
17682024-05-08T03:33:32.191ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17692024-05-08T03:33:32.191ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
17702024-05-08T03:33:32.210ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17712024-05-08T03:33:32.210ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
17722024-05-08T03:33:32.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17732024-05-08T03:33:32.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
17742024-05-08T03:33:32.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17752024-05-08T03:33:32.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
17762024-05-08T03:33:32.267ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17772024-05-08T03:33:32.267ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
17782024-05-08T03:33:32.284ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17792024-05-08T03:33:32.284ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17802024-05-08T03:33:32.301ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17812024-05-08T03:33:32.301ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
17822024-05-08T03:33:32.320ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17832024-05-08T03:33:32.320ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
17842024-05-08T03:33:32.320ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
17852024-05-08T03:33:32.320ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17862024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
17872024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
17882024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
17892024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
17902024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
17912024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
17922024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
17932024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
17942024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
17952024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
17962024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
17972024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
17982024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
17992024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
18002024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
18012024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
18022024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
18032024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
18042024-05-08T03:33:32.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
18052024-05-08T03:33:32.416ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 473473 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 3a37163b-2356-4357-b900-e9d8bc3bd204 response_code = 500 uri = /omicron-zones
18062024-05-08T03:33:32.416ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3a37163b-2356-4357-b900-e9d8bc3bd204", "content-length": "124", "date": "Wed, 08 May 2024 03:33:31 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3a37163b-2356-4357-b900-e9d8bc3bd204" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 958.491049ms sled_address = [fd00:1122:3344:101::1]:12345
18072024-05-08T03:33:33.376ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
18082024-05-08T03:33:33.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18092024-05-08T03:33:33.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18102024-05-08T03:33:33.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18112024-05-08T03:33:33.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
18122024-05-08T03:33:33.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18132024-05-08T03:33:33.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18142024-05-08T03:33:33.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18152024-05-08T03:33:33.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18162024-05-08T03:33:33.458ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18172024-05-08T03:33:33.458ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18182024-05-08T03:33:33.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18192024-05-08T03:33:33.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18202024-05-08T03:33:33.496ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18212024-05-08T03:33:33.496ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18222024-05-08T03:33:33.516ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18232024-05-08T03:33:33.516ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18242024-05-08T03:33:33.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18252024-05-08T03:33:33.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18262024-05-08T03:33:33.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18272024-05-08T03:33:33.555ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18282024-05-08T03:33:33.575ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
18292024-05-08T03:33:33.575ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18302024-05-08T03:33:33.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18312024-05-08T03:33:33.595ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
18322024-05-08T03:33:33.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18332024-05-08T03:33:33.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
18342024-05-08T03:33:33.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18352024-05-08T03:33:33.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
18362024-05-08T03:33:33.651ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18372024-05-08T03:33:33.651ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
18382024-05-08T03:33:33.669ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18392024-05-08T03:33:33.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
18402024-05-08T03:33:33.689ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18412024-05-08T03:33:33.689ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
18422024-05-08T03:33:33.707ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18432024-05-08T03:33:33.707ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
18442024-05-08T03:33:33.728ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18452024-05-08T03:33:33.728ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
18462024-05-08T03:33:33.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18472024-05-08T03:33:33.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
18482024-05-08T03:33:33.769ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18492024-05-08T03:33:33.769ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
18502024-05-08T03:33:33.769ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
18512024-05-08T03:33:33.769ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18522024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
18532024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
18542024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
18552024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
18562024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
18572024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
18582024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
18592024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
18602024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
18612024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
18622024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
18632024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
18642024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
18652024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
18662024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
18672024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
18682024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
18692024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
18702024-05-08T03:33:33.872ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
18712024-05-08T03:33:33.872ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 496238 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = e0aeaf03-4975-4752-a3bf-754a88491deb response_code = 500 uri = /omicron-zones
18722024-05-08T03:33:33.872ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e0aeaf03-4975-4752-a3bf-754a88491deb", "content-length": "124", "date": "Wed, 08 May 2024 03:33:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e0aeaf03-4975-4752-a3bf-754a88491deb" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 2.516912722s sled_address = [fd00:1122:3344:101::1]:12345
18732024-05-08T03:33:36.391ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
18742024-05-08T03:33:36.392ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18752024-05-08T03:33:36.392ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18762024-05-08T03:33:36.415ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18772024-05-08T03:33:36.415ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
18782024-05-08T03:33:36.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18792024-05-08T03:33:36.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18802024-05-08T03:33:36.457ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18812024-05-08T03:33:36.457ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18822024-05-08T03:33:36.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18832024-05-08T03:33:36.475ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18842024-05-08T03:33:36.494ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18852024-05-08T03:33:36.494ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18862024-05-08T03:33:36.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18872024-05-08T03:33:36.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18882024-05-08T03:33:36.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18892024-05-08T03:33:36.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18902024-05-08T03:33:36.552ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18912024-05-08T03:33:36.552ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18922024-05-08T03:33:36.572ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18932024-05-08T03:33:36.572ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18942024-05-08T03:33:36.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
18952024-05-08T03:33:36.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18962024-05-08T03:33:36.611ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18972024-05-08T03:33:36.611ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
18982024-05-08T03:33:36.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18992024-05-08T03:33:36.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
19002024-05-08T03:33:36.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19012024-05-08T03:33:36.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
19022024-05-08T03:33:36.669ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19032024-05-08T03:33:36.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
19042024-05-08T03:33:36.689ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19052024-05-08T03:33:36.689ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
19062024-05-08T03:33:36.707ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19072024-05-08T03:33:36.707ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
19082024-05-08T03:33:36.726ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19092024-05-08T03:33:36.726ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
19102024-05-08T03:33:36.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19112024-05-08T03:33:36.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
19122024-05-08T03:33:36.766ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19132024-05-08T03:33:36.766ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
19142024-05-08T03:33:36.785ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19152024-05-08T03:33:36.785ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
19162024-05-08T03:33:36.785ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
19172024-05-08T03:33:36.786ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19182024-05-08T03:33:36.844ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41489
19192024-05-08T03:33:36.844ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 170 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41489 req_id = 6f346d02-ddc4-40ad-b45d-404076dd62da response_code = 200 uri = /baseboard
19202024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
19212024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
19222024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
19232024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
19242024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
19252024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
19262024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
19272024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
19282024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
19292024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
19302024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
19312024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
19322024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
19332024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
19342024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
19352024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
19362024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
19372024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
19382024-05-08T03:33:36.882ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
19392024-05-08T03:33:36.883ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 491034 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = f382cc51-0951-42dc-a676-a69bec7be12b response_code = 500 uri = /omicron-zones
19402024-05-08T03:33:36.883ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f382cc51-0951-42dc-a676-a69bec7be12b", "content-length": "124", "date": "Wed, 08 May 2024 03:33:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f382cc51-0951-42dc-a676-a69bec7be12b" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 1.842046584s sled_address = [fd00:1122:3344:101::1]:12345
19412024-05-08T03:33:38.284ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.486963091s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
19422024-05-08T03:33:38.726ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
19432024-05-08T03:33:38.726ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19442024-05-08T03:33:38.726ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19452024-05-08T03:33:38.750ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19462024-05-08T03:33:38.750ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
19472024-05-08T03:33:38.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19482024-05-08T03:33:38.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19492024-05-08T03:33:38.791ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19502024-05-08T03:33:38.791ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19512024-05-08T03:33:38.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19522024-05-08T03:33:38.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19532024-05-08T03:33:38.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19542024-05-08T03:33:38.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19552024-05-08T03:33:38.852ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19562024-05-08T03:33:38.852ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19572024-05-08T03:33:38.873ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19582024-05-08T03:33:38.873ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19592024-05-08T03:33:38.892ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19602024-05-08T03:33:38.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19612024-05-08T03:33:38.915ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19622024-05-08T03:33:38.915ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19632024-05-08T03:33:38.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
19642024-05-08T03:33:38.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19652024-05-08T03:33:38.958ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19662024-05-08T03:33:38.959ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
19672024-05-08T03:33:38.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19682024-05-08T03:33:38.978ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
19692024-05-08T03:33:38.998ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19702024-05-08T03:33:38.998ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
19712024-05-08T03:33:39.018ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19722024-05-08T03:33:39.018ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
19732024-05-08T03:33:39.039ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19742024-05-08T03:33:39.039ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
19752024-05-08T03:33:39.059ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19762024-05-08T03:33:39.059ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
19772024-05-08T03:33:39.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19782024-05-08T03:33:39.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
19792024-05-08T03:33:39.100ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19802024-05-08T03:33:39.100ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
19812024-05-08T03:33:39.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19822024-05-08T03:33:39.120ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
19832024-05-08T03:33:39.142ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19842024-05-08T03:33:39.142ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
19852024-05-08T03:33:39.142ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
19862024-05-08T03:33:39.142ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19872024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
19882024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
19892024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
19902024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
19912024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
19922024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
19932024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
19942024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
19952024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
19962024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
19972024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
19982024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
19992024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
20002024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
20012024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
20022024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
20032024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
20042024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
20052024-05-08T03:33:39.237ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
20062024-05-08T03:33:39.237ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 511265 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = f7999682-9b73-4220-9e10-17ba0f9292b2 response_code = 500 uri = /omicron-zones
20072024-05-08T03:33:39.238ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f7999682-9b73-4220-9e10-17ba0f9292b2", "content-length": "124", "date": "Wed, 08 May 2024 03:33:38 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f7999682-9b73-4220-9e10-17ba0f9292b2" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 3.667256727s sled_address = [fd00:1122:3344:101::1]:12345
20082024-05-08T03:33:42.906ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
20092024-05-08T03:33:42.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20102024-05-08T03:33:42.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20112024-05-08T03:33:42.931ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20122024-05-08T03:33:42.931ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
20132024-05-08T03:33:42.954ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20142024-05-08T03:33:42.954ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20152024-05-08T03:33:42.976ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20162024-05-08T03:33:42.976ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20172024-05-08T03:33:42.996ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20182024-05-08T03:33:42.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20192024-05-08T03:33:43.018ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20202024-05-08T03:33:43.018ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20212024-05-08T03:33:43.038ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20222024-05-08T03:33:43.038ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20232024-05-08T03:33:43.060ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20242024-05-08T03:33:43.060ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20252024-05-08T03:33:43.081ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20262024-05-08T03:33:43.081ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20272024-05-08T03:33:43.103ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20282024-05-08T03:33:43.103ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20292024-05-08T03:33:43.126ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
20302024-05-08T03:33:43.126ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20312024-05-08T03:33:43.148ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20322024-05-08T03:33:43.148ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
20332024-05-08T03:33:43.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20342024-05-08T03:33:43.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
20352024-05-08T03:33:43.192ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20362024-05-08T03:33:43.192ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
20372024-05-08T03:33:43.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20382024-05-08T03:33:43.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
20392024-05-08T03:33:43.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20402024-05-08T03:33:43.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
20412024-05-08T03:33:43.259ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20422024-05-08T03:33:43.259ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
20432024-05-08T03:33:43.281ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20442024-05-08T03:33:43.281ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
20452024-05-08T03:33:43.303ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20462024-05-08T03:33:43.303ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
20472024-05-08T03:33:43.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20482024-05-08T03:33:43.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
20492024-05-08T03:33:43.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20502024-05-08T03:33:43.346ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
20512024-05-08T03:33:43.346ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
20522024-05-08T03:33:43.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20532024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
20542024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
20552024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
20562024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
20572024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
20582024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
20592024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
20602024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
20612024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
20622024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
20632024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
20642024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
20652024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
20662024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
20672024-05-08T03:33:43.451ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
20682024-05-08T03:33:43.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
20692024-05-08T03:33:43.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
20702024-05-08T03:33:43.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
20712024-05-08T03:33:43.452ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
20722024-05-08T03:33:43.452ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 545337 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 17500081-3dfb-4f58-b1c4-18c2d2f699a7 response_code = 500 uri = /omicron-zones
20732024-05-08T03:33:43.452ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "17500081-3dfb-4f58-b1c4-18c2d2f699a7", "content-length": "124", "date": "Wed, 08 May 2024 03:33:42 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "17500081-3dfb-4f58-b1c4-18c2d2f699a7" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 4.58136794s sled_address = [fd00:1122:3344:101::1]:12345
20742024-05-08T03:33:48.035ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
20752024-05-08T03:33:48.037ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20762024-05-08T03:33:48.037ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20772024-05-08T03:33:48.059ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20782024-05-08T03:33:48.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
20792024-05-08T03:33:48.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20802024-05-08T03:33:48.077ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20812024-05-08T03:33:48.099ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20822024-05-08T03:33:48.099ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20832024-05-08T03:33:48.117ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20842024-05-08T03:33:48.117ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20852024-05-08T03:33:48.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20862024-05-08T03:33:48.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20872024-05-08T03:33:48.154ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20882024-05-08T03:33:48.154ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20892024-05-08T03:33:48.173ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20902024-05-08T03:33:48.173ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20912024-05-08T03:33:48.191ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20922024-05-08T03:33:48.191ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20932024-05-08T03:33:48.209ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20942024-05-08T03:33:48.209ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20952024-05-08T03:33:48.228ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
20962024-05-08T03:33:48.228ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20972024-05-08T03:33:48.246ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20982024-05-08T03:33:48.246ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
20992024-05-08T03:33:48.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21002024-05-08T03:33:48.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
21012024-05-08T03:33:48.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21022024-05-08T03:33:48.283ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
21032024-05-08T03:33:48.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21042024-05-08T03:33:48.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
21052024-05-08T03:33:48.320ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21062024-05-08T03:33:48.320ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
21072024-05-08T03:33:48.336ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21082024-05-08T03:33:48.336ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
21092024-05-08T03:33:48.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21102024-05-08T03:33:48.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
21112024-05-08T03:33:48.372ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21122024-05-08T03:33:48.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
21132024-05-08T03:33:48.392ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21142024-05-08T03:33:48.392ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
21152024-05-08T03:33:48.412ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21162024-05-08T03:33:48.412ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
21172024-05-08T03:33:48.412ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
21182024-05-08T03:33:48.412ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21192024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
21202024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
21212024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
21222024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
21232024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
21242024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
21252024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
21262024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
21272024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
21282024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
21292024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
21302024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
21312024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
21322024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
21332024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
21342024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
21352024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
21362024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
21372024-05-08T03:33:48.505ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
21382024-05-08T03:33:48.506ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 469777 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 3fc2a4c6-aafc-493e-954b-0d11b937a9a7 response_code = 500 uri = /omicron-zones
21392024-05-08T03:33:48.506ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3fc2a4c6-aafc-493e-954b-0d11b937a9a7", "content-length": "124", "date": "Wed, 08 May 2024 03:33:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3fc2a4c6-aafc-493e-954b-0d11b937a9a7" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 2.545782529s sled_address = [fd00:1122:3344:101::1]:12345
21402024-05-08T03:33:51.053ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
21412024-05-08T03:33:51.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21422024-05-08T03:33:51.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21432024-05-08T03:33:51.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21442024-05-08T03:33:51.075ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
21452024-05-08T03:33:51.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21462024-05-08T03:33:51.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21472024-05-08T03:33:51.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21482024-05-08T03:33:51.112ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21492024-05-08T03:33:51.129ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21502024-05-08T03:33:51.129ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21512024-05-08T03:33:51.148ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21522024-05-08T03:33:51.148ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21532024-05-08T03:33:51.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21542024-05-08T03:33:51.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21552024-05-08T03:33:51.183ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21562024-05-08T03:33:51.183ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21572024-05-08T03:33:51.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21582024-05-08T03:33:51.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21592024-05-08T03:33:51.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21602024-05-08T03:33:51.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21612024-05-08T03:33:51.238ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
21622024-05-08T03:33:51.238ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21632024-05-08T03:33:51.257ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21642024-05-08T03:33:51.257ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
21652024-05-08T03:33:51.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21662024-05-08T03:33:51.274ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
21672024-05-08T03:33:51.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21682024-05-08T03:33:51.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
21692024-05-08T03:33:51.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21702024-05-08T03:33:51.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
21712024-05-08T03:33:51.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21722024-05-08T03:33:51.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
21732024-05-08T03:33:51.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21742024-05-08T03:33:51.347ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
21752024-05-08T03:33:51.363ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21762024-05-08T03:33:51.363ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
21772024-05-08T03:33:51.381ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21782024-05-08T03:33:51.381ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
21792024-05-08T03:33:51.399ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21802024-05-08T03:33:51.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
21812024-05-08T03:33:51.417ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21822024-05-08T03:33:51.417ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
21832024-05-08T03:33:51.417ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
21842024-05-08T03:33:51.417ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21852024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
21862024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
21872024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
21882024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
21892024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
21902024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
21912024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
21922024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
21932024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
21942024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
21952024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
21962024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
21972024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
21982024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
21992024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
22002024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
22012024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
22022024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
22032024-05-08T03:33:51.504ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
22042024-05-08T03:33:51.504ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 450611 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = c0d08d35-7451-4e83-80af-a60387e84499 response_code = 500 uri = /omicron-zones
22052024-05-08T03:33:51.504ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c0d08d35-7451-4e83-80af-a60387e84499", "content-length": "124", "date": "Wed, 08 May 2024 03:33:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c0d08d35-7451-4e83-80af-a60387e84499" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 6.61461392s sled_address = [fd00:1122:3344:101::1]:12345
22062024-05-08T03:33:55.809ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.102571867s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
22072024-05-08T03:33:58.120ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
22082024-05-08T03:33:58.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22092024-05-08T03:33:58.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22102024-05-08T03:33:58.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22112024-05-08T03:33:58.144ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
22122024-05-08T03:33:58.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22132024-05-08T03:33:58.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22142024-05-08T03:33:58.179ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22152024-05-08T03:33:58.179ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22162024-05-08T03:33:58.198ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22172024-05-08T03:33:58.200ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22182024-05-08T03:33:58.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22192024-05-08T03:33:58.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22202024-05-08T03:33:58.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22212024-05-08T03:33:58.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22222024-05-08T03:33:58.252ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22232024-05-08T03:33:58.252ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22242024-05-08T03:33:58.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22252024-05-08T03:33:58.271ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22262024-05-08T03:33:58.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22272024-05-08T03:33:58.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22282024-05-08T03:33:58.307ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
22292024-05-08T03:33:58.307ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22302024-05-08T03:33:58.326ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22312024-05-08T03:33:58.326ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
22322024-05-08T03:33:58.345ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22332024-05-08T03:33:58.345ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
22342024-05-08T03:33:58.363ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22352024-05-08T03:33:58.363ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
22362024-05-08T03:33:58.382ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22372024-05-08T03:33:58.382ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
22382024-05-08T03:33:58.402ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22392024-05-08T03:33:58.402ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
22402024-05-08T03:33:58.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22412024-05-08T03:33:58.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
22422024-05-08T03:33:58.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22432024-05-08T03:33:58.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
22442024-05-08T03:33:58.457ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22452024-05-08T03:33:58.457ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
22462024-05-08T03:33:58.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22472024-05-08T03:33:58.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
22482024-05-08T03:33:58.495ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22492024-05-08T03:33:58.496ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
22502024-05-08T03:33:58.496ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
22512024-05-08T03:33:58.496ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22522024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
22532024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
22542024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
22552024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
22562024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
22572024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
22582024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
22592024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
22602024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
22612024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
22622024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
22632024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
22642024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
22652024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
22662024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
22672024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
22682024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
22692024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
22702024-05-08T03:33:58.589ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
22712024-05-08T03:33:58.589ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 468477 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = e7273ff0-c9a1-4ec4-88b0-751876065178 response_code = 500 uri = /omicron-zones
22722024-05-08T03:33:58.589ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e7273ff0-c9a1-4ec4-88b0-751876065178", "content-length": "124", "date": "Wed, 08 May 2024 03:33:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e7273ff0-c9a1-4ec4-88b0-751876065178" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 3.855983128s sled_address = [fd00:1122:3344:101::1]:12345
22732024-05-08T03:34:02.448ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
22742024-05-08T03:34:02.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22752024-05-08T03:34:02.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22762024-05-08T03:34:02.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22772024-05-08T03:34:02.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
22782024-05-08T03:34:02.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22792024-05-08T03:34:02.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22802024-05-08T03:34:02.510ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22812024-05-08T03:34:02.510ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22822024-05-08T03:34:02.529ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22832024-05-08T03:34:02.529ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22842024-05-08T03:34:02.548ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22852024-05-08T03:34:02.548ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22862024-05-08T03:34:02.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22872024-05-08T03:34:02.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22882024-05-08T03:34:02.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22892024-05-08T03:34:02.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22902024-05-08T03:34:02.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22912024-05-08T03:34:02.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22922024-05-08T03:34:02.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22932024-05-08T03:34:02.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22942024-05-08T03:34:02.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
22952024-05-08T03:34:02.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22962024-05-08T03:34:02.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22972024-05-08T03:34:02.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
22982024-05-08T03:34:02.677ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22992024-05-08T03:34:02.677ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
23002024-05-08T03:34:02.696ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23012024-05-08T03:34:02.696ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
23022024-05-08T03:34:02.715ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23032024-05-08T03:34:02.715ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
23042024-05-08T03:34:02.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23052024-05-08T03:34:02.734ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
23062024-05-08T03:34:02.753ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23072024-05-08T03:34:02.753ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
23082024-05-08T03:34:02.772ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23092024-05-08T03:34:02.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
23102024-05-08T03:34:02.791ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23112024-05-08T03:34:02.791ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
23122024-05-08T03:34:02.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23132024-05-08T03:34:02.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
23142024-05-08T03:34:02.829ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
23152024-05-08T03:34:02.829ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
23162024-05-08T03:34:02.829ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
23172024-05-08T03:34:02.829ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
23182024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
23192024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
23202024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
23212024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
23222024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
23232024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
23242024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
23252024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
23262024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
23272024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
23282024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
23292024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
23302024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
23312024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
23322024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
23332024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
23342024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
23352024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
23362024-05-08T03:34:02.923ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
23372024-05-08T03:34:02.923ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 475024 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 88e6d25e-dc1a-4b92-9e37-3fafdf727e1f response_code = 500 uri = /omicron-zones
23382024-05-08T03:34:02.923ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "88e6d25e-dc1a-4b92-9e37-3fafdf727e1f", "content-length": "124", "date": "Wed, 08 May 2024 03:34:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "88e6d25e-dc1a-4b92-9e37-3fafdf727e1f" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 3.457766996s sled_address = [fd00:1122:3344:101::1]:12345
23392024-05-08T03:34:06.382ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
23402024-05-08T03:34:06.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23412024-05-08T03:34:06.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23422024-05-08T03:34:06.407ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23432024-05-08T03:34:06.407ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
23442024-05-08T03:34:06.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23452024-05-08T03:34:06.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23462024-05-08T03:34:06.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23472024-05-08T03:34:06.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23482024-05-08T03:34:06.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23492024-05-08T03:34:06.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23502024-05-08T03:34:06.480ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23512024-05-08T03:34:06.480ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23522024-05-08T03:34:06.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23532024-05-08T03:34:06.499ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23542024-05-08T03:34:06.516ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23552024-05-08T03:34:06.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23562024-05-08T03:34:06.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23572024-05-08T03:34:06.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23582024-05-08T03:34:06.553ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23592024-05-08T03:34:06.553ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23602024-05-08T03:34:06.572ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
23612024-05-08T03:34:06.572ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
23622024-05-08T03:34:06.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23632024-05-08T03:34:06.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
23642024-05-08T03:34:06.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23652024-05-08T03:34:06.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
23662024-05-08T03:34:06.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23672024-05-08T03:34:06.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
23682024-05-08T03:34:06.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23692024-05-08T03:34:06.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
23702024-05-08T03:34:06.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23712024-05-08T03:34:06.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
23722024-05-08T03:34:06.682ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23732024-05-08T03:34:06.682ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
23742024-05-08T03:34:06.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23752024-05-08T03:34:06.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
23762024-05-08T03:34:06.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23772024-05-08T03:34:06.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
23782024-05-08T03:34:06.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23792024-05-08T03:34:06.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
23802024-05-08T03:34:06.753ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
23812024-05-08T03:34:06.753ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
23822024-05-08T03:34:06.753ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
23832024-05-08T03:34:06.753ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
23842024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
23852024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
23862024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
23872024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
23882024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
23892024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
23902024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
23912024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
23922024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
23932024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
23942024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
23952024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
23962024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
23972024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
23982024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
23992024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
24002024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
24012024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
24022024-05-08T03:34:06.843ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
24032024-05-08T03:34:06.844ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 461077 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = aa8e1afc-9a0a-455b-aa92-096901529541 response_code = 500 uri = /omicron-zones
24042024-05-08T03:34:06.844ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "aa8e1afc-9a0a-455b-aa92-096901529541", "content-length": "124", "date": "Wed, 08 May 2024 03:34:06 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "aa8e1afc-9a0a-455b-aa92-096901529541" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 7.017581671s sled_address = [fd00:1122:3344:101::1]:12345
24052024-05-08T03:34:06.879ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44188
24062024-05-08T03:34:06.879ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 52 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44188 req_id = bcd0c2a2-793e-4d2f-b448-393ae301f44c response_code = 200 uri = /baseboard
24072024-05-08T03:34:13.864ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
24082024-05-08T03:34:13.865ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24092024-05-08T03:34:13.865ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24102024-05-08T03:34:13.887ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24112024-05-08T03:34:13.887ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
24122024-05-08T03:34:13.908ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24132024-05-08T03:34:13.908ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24142024-05-08T03:34:13.930ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24152024-05-08T03:34:13.930ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24162024-05-08T03:34:13.950ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24172024-05-08T03:34:13.950ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24182024-05-08T03:34:13.957ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 7.432261137s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
24192024-05-08T03:34:13.971ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24202024-05-08T03:34:13.971ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24212024-05-08T03:34:13.992ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24222024-05-08T03:34:13.992ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24232024-05-08T03:34:14.013ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24242024-05-08T03:34:14.013ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24252024-05-08T03:34:14.035ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24262024-05-08T03:34:14.035ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24272024-05-08T03:34:14.058ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24282024-05-08T03:34:14.058ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24292024-05-08T03:34:14.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
24302024-05-08T03:34:14.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24312024-05-08T03:34:14.100ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24322024-05-08T03:34:14.100ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
24332024-05-08T03:34:14.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24342024-05-08T03:34:14.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
24352024-05-08T03:34:14.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24362024-05-08T03:34:14.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
24372024-05-08T03:34:14.162ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24382024-05-08T03:34:14.162ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
24392024-05-08T03:34:14.182ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24402024-05-08T03:34:14.182ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
24412024-05-08T03:34:14.204ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24422024-05-08T03:34:14.204ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
24432024-05-08T03:34:14.225ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24442024-05-08T03:34:14.225ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
24452024-05-08T03:34:14.243ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24462024-05-08T03:34:14.243ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
24472024-05-08T03:34:14.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24482024-05-08T03:34:14.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
24492024-05-08T03:34:14.284ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24502024-05-08T03:34:14.284ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
24512024-05-08T03:34:14.285ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
24522024-05-08T03:34:14.285ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24532024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
24542024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
24552024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
24562024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
24572024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
24582024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
24592024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
24602024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
24612024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
24622024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
24632024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
24642024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
24652024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
24662024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
24672024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
24682024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
24692024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
24702024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
24712024-05-08T03:34:14.387ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
24722024-05-08T03:34:14.387ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 522856 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = b4a7b511-621b-402b-b235-9fc2d0a6333f response_code = 500 uri = /omicron-zones
24732024-05-08T03:34:14.388ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b4a7b511-621b-402b-b235-9fc2d0a6333f", "content-length": "124", "date": "Wed, 08 May 2024 03:34:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b4a7b511-621b-402b-b235-9fc2d0a6333f" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 12.894142905s sled_address = [fd00:1122:3344:101::1]:12345
24742024-05-08T03:34:27.286ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
24752024-05-08T03:34:27.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24762024-05-08T03:34:27.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24772024-05-08T03:34:27.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24782024-05-08T03:34:27.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
24792024-05-08T03:34:27.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24802024-05-08T03:34:27.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24812024-05-08T03:34:27.351ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24822024-05-08T03:34:27.351ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24832024-05-08T03:34:27.369ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24842024-05-08T03:34:27.369ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24852024-05-08T03:34:27.386ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24862024-05-08T03:34:27.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24872024-05-08T03:34:27.404ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24882024-05-08T03:34:27.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24892024-05-08T03:34:27.423ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24902024-05-08T03:34:27.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24912024-05-08T03:34:27.442ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24922024-05-08T03:34:27.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24932024-05-08T03:34:27.461ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24942024-05-08T03:34:27.461ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24952024-05-08T03:34:27.480ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
24962024-05-08T03:34:27.480ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24972024-05-08T03:34:27.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24982024-05-08T03:34:27.498ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
24992024-05-08T03:34:27.517ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25002024-05-08T03:34:27.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
25012024-05-08T03:34:27.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25022024-05-08T03:34:27.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
25032024-05-08T03:34:27.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25042024-05-08T03:34:27.552ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
25052024-05-08T03:34:27.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25062024-05-08T03:34:27.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
25072024-05-08T03:34:27.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25082024-05-08T03:34:27.588ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
25092024-05-08T03:34:27.605ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25102024-05-08T03:34:27.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
25112024-05-08T03:34:27.622ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25122024-05-08T03:34:27.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
25132024-05-08T03:34:27.640ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25142024-05-08T03:34:27.640ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
25152024-05-08T03:34:27.658ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25162024-05-08T03:34:27.658ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
25172024-05-08T03:34:27.658ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
25182024-05-08T03:34:27.659ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25192024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
25202024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
25212024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
25222024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
25232024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
25242024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
25252024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
25262024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
25272024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
25282024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
25292024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
25302024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
25312024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
25322024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
25332024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
25342024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
25352024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
25362024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
25372024-05-08T03:34:27.751ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
25382024-05-08T03:34:27.751ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 463980 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = d2a35c31-9d39-4f5a-9c4f-6e03f067c475 response_code = 500 uri = /omicron-zones
25392024-05-08T03:34:27.751ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d2a35c31-9d39-4f5a-9c4f-6e03f067c475", "content-length": "124", "date": "Wed, 08 May 2024 03:34:27 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d2a35c31-9d39-4f5a-9c4f-6e03f067c475" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 9.91621435s sled_address = [fd00:1122:3344:101::1]:12345
25402024-05-08T03:34:36.433ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 16.079319507s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
25412024-05-08T03:34:36.920ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33305
25422024-05-08T03:34:36.920ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 50 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33305 req_id = 7ed847fa-7aea-4360-a92f-8266677ed6cb response_code = 200 uri = /baseboard
25432024-05-08T03:34:37.671ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
25442024-05-08T03:34:37.671ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25452024-05-08T03:34:37.671ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25462024-05-08T03:34:37.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25472024-05-08T03:34:37.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
25482024-05-08T03:34:37.712ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25492024-05-08T03:34:37.715ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25502024-05-08T03:34:37.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25512024-05-08T03:34:37.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25522024-05-08T03:34:37.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25532024-05-08T03:34:37.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25542024-05-08T03:34:37.767ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25552024-05-08T03:34:37.767ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25562024-05-08T03:34:37.785ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25572024-05-08T03:34:37.785ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25582024-05-08T03:34:37.803ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25592024-05-08T03:34:37.803ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25602024-05-08T03:34:37.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25612024-05-08T03:34:37.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25622024-05-08T03:34:37.839ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25632024-05-08T03:34:37.839ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25642024-05-08T03:34:37.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
25652024-05-08T03:34:37.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
25662024-05-08T03:34:37.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25672024-05-08T03:34:37.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
25682024-05-08T03:34:37.895ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25692024-05-08T03:34:37.895ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
25702024-05-08T03:34:37.912ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25712024-05-08T03:34:37.912ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
25722024-05-08T03:34:37.930ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25732024-05-08T03:34:37.930ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
25742024-05-08T03:34:37.950ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25752024-05-08T03:34:37.950ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
25762024-05-08T03:34:37.967ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25772024-05-08T03:34:37.967ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
25782024-05-08T03:34:37.985ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25792024-05-08T03:34:37.985ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
25802024-05-08T03:34:38.002ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25812024-05-08T03:34:38.002ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
25822024-05-08T03:34:38.021ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25832024-05-08T03:34:38.021ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
25842024-05-08T03:34:38.039ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25852024-05-08T03:34:38.039ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
25862024-05-08T03:34:38.040ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
25872024-05-08T03:34:38.040ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25882024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
25892024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
25902024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
25912024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
25922024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
25932024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
25942024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
25952024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
25962024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
25972024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
25982024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
25992024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
26002024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
26012024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
26022024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
26032024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
26042024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
26052024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
26062024-05-08T03:34:38.133ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
26072024-05-08T03:34:38.133ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 462119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 0bb8d76d-8b6d-493d-b14d-eca623f9b2a4 response_code = 500 uri = /omicron-zones
26082024-05-08T03:34:38.134ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0bb8d76d-8b6d-493d-b14d-eca623f9b2a4", "content-length": "124", "date": "Wed, 08 May 2024 03:34:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0bb8d76d-8b6d-493d-b14d-eca623f9b2a4" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 10.3525632s sled_address = [fd00:1122:3344:101::1]:12345
26092024-05-08T03:34:48.490ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
26102024-05-08T03:34:48.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26112024-05-08T03:34:48.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26122024-05-08T03:34:48.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26132024-05-08T03:34:48.515ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
26142024-05-08T03:34:48.534ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26152024-05-08T03:34:48.534ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26162024-05-08T03:34:48.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26172024-05-08T03:34:48.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26182024-05-08T03:34:48.571ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26192024-05-08T03:34:48.571ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26202024-05-08T03:34:48.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26212024-05-08T03:34:48.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26222024-05-08T03:34:48.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26232024-05-08T03:34:48.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26242024-05-08T03:34:48.626ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26252024-05-08T03:34:48.626ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26262024-05-08T03:34:48.644ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26272024-05-08T03:34:48.644ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26282024-05-08T03:34:48.661ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26292024-05-08T03:34:48.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26302024-05-08T03:34:48.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
26312024-05-08T03:34:48.681ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
26322024-05-08T03:34:48.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26332024-05-08T03:34:48.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
26342024-05-08T03:34:48.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26352024-05-08T03:34:48.718ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
26362024-05-08T03:34:48.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26372024-05-08T03:34:48.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
26382024-05-08T03:34:48.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26392024-05-08T03:34:48.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
26402024-05-08T03:34:48.772ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26412024-05-08T03:34:48.772ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
26422024-05-08T03:34:48.790ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26432024-05-08T03:34:48.790ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
26442024-05-08T03:34:48.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26452024-05-08T03:34:48.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
26462024-05-08T03:34:48.827ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26472024-05-08T03:34:48.827ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
26482024-05-08T03:34:48.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26492024-05-08T03:34:48.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
26502024-05-08T03:34:48.864ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26512024-05-08T03:34:48.864ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
26522024-05-08T03:34:48.864ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
26532024-05-08T03:34:48.864ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26542024-05-08T03:34:48.956ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
26552024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
26562024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
26572024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
26582024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
26592024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
26602024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
26612024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
26622024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
26632024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
26642024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
26652024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
26662024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
26672024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
26682024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
26692024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
26702024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
26712024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
26722024-05-08T03:34:48.957ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
26732024-05-08T03:34:48.957ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 466259 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = ecc80f57-8490-47db-ae40-086e4f7f6d18 response_code = 500 uri = /omicron-zones
26742024-05-08T03:34:48.957ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ecc80f57-8490-47db-ae40-086e4f7f6d18", "content-length": "124", "date": "Wed, 08 May 2024 03:34:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ecc80f57-8490-47db-ae40-086e4f7f6d18" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 18.949901779s sled_address = [fd00:1122:3344:101::1]:12345
26752024-05-08T03:35:06.962ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54615
26762024-05-08T03:35:06.962ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54615 req_id = 3918c266-3822-41db-a03b-2a03103cf991 response_code = 200 uri = /baseboard
26772024-05-08T03:35:07.555ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 40.247931112s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
26782024-05-08T03:35:07.913ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
26792024-05-08T03:35:07.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26802024-05-08T03:35:07.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26812024-05-08T03:35:07.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26822024-05-08T03:35:07.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
26832024-05-08T03:35:07.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26842024-05-08T03:35:07.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26852024-05-08T03:35:07.975ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26862024-05-08T03:35:07.975ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26872024-05-08T03:35:07.994ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26882024-05-08T03:35:07.994ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26892024-05-08T03:35:08.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26902024-05-08T03:35:08.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26912024-05-08T03:35:08.033ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26922024-05-08T03:35:08.035ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26932024-05-08T03:35:08.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26942024-05-08T03:35:08.052ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26952024-05-08T03:35:08.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26962024-05-08T03:35:08.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26972024-05-08T03:35:08.087ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26982024-05-08T03:35:08.087ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26992024-05-08T03:35:08.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
27002024-05-08T03:35:08.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
27012024-05-08T03:35:08.125ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27022024-05-08T03:35:08.125ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
27032024-05-08T03:35:08.142ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27042024-05-08T03:35:08.142ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
27052024-05-08T03:35:08.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27062024-05-08T03:35:08.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
27072024-05-08T03:35:08.178ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27082024-05-08T03:35:08.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
27092024-05-08T03:35:08.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27102024-05-08T03:35:08.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
27112024-05-08T03:35:08.214ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27122024-05-08T03:35:08.214ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
27132024-05-08T03:35:08.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27142024-05-08T03:35:08.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
27152024-05-08T03:35:08.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27162024-05-08T03:35:08.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
27172024-05-08T03:35:08.269ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27182024-05-08T03:35:08.269ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
27192024-05-08T03:35:08.285ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
27202024-05-08T03:35:08.285ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
27212024-05-08T03:35:08.286ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
27222024-05-08T03:35:08.286ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
27232024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
27242024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
27252024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
27262024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
27272024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
27282024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
27292024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
27302024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
27312024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
27322024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
27332024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
27342024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
27352024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
27362024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
27372024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
27382024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
27392024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
27402024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
27412024-05-08T03:35:08.375ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
27422024-05-08T03:35:08.375ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 462032 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 46ff6853-869f-4429-bd24-2b89e5e99288 response_code = 500 uri = /omicron-zones
27432024-05-08T03:35:08.375ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "46ff6853-869f-4429-bd24-2b89e5e99288", "content-length": "124", "date": "Wed, 08 May 2024 03:35:07 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "46ff6853-869f-4429-bd24-2b89e5e99288" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 27.278658679s sled_address = [fd00:1122:3344:101::1]:12345
27442024-05-08T03:35:35.660ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
27452024-05-08T03:35:35.661ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27462024-05-08T03:35:35.661ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27472024-05-08T03:35:35.682ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27482024-05-08T03:35:35.682ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
27492024-05-08T03:35:35.701ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27502024-05-08T03:35:35.703ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27512024-05-08T03:35:35.720ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27522024-05-08T03:35:35.720ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27532024-05-08T03:35:35.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27542024-05-08T03:35:35.741ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27552024-05-08T03:35:35.759ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27562024-05-08T03:35:35.759ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27572024-05-08T03:35:35.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27582024-05-08T03:35:35.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27592024-05-08T03:35:35.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27602024-05-08T03:35:35.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27612024-05-08T03:35:35.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27622024-05-08T03:35:35.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27632024-05-08T03:35:35.831ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27642024-05-08T03:35:35.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27652024-05-08T03:35:35.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
27662024-05-08T03:35:35.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
27672024-05-08T03:35:35.868ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27682024-05-08T03:35:35.868ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
27692024-05-08T03:35:35.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27702024-05-08T03:35:35.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
27712024-05-08T03:35:35.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27722024-05-08T03:35:35.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
27732024-05-08T03:35:35.922ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27742024-05-08T03:35:35.922ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
27752024-05-08T03:35:35.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27762024-05-08T03:35:35.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
27772024-05-08T03:35:35.958ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27782024-05-08T03:35:35.958ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
27792024-05-08T03:35:35.978ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27802024-05-08T03:35:35.978ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
27812024-05-08T03:35:35.996ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27822024-05-08T03:35:35.996ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
27832024-05-08T03:35:36.015ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27842024-05-08T03:35:36.015ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
27852024-05-08T03:35:36.033ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
27862024-05-08T03:35:36.033ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
27872024-05-08T03:35:36.033ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
27882024-05-08T03:35:36.033ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
27892024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
27902024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
27912024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
27922024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
27932024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
27942024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
27952024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
27962024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
27972024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
27982024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
27992024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
28002024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
28012024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
28022024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
28032024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
28042024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
28052024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
28062024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
28072024-05-08T03:35:36.127ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3030 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
28082024-05-08T03:35:36.128ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3", TimeNotSynchronized), ("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5", TimeNotSynchronized), ("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d", TimeNotSynchronized), ("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327", TimeNotSynchronized), ("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2", TimeNotSynchronized), ("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817", TimeNotSynchronized), ("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818", TimeNotSynchronized), ("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4", TimeNotSynchronized), ("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a", TimeNotSynchronized), ("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd", TimeNotSynchronized), ("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb", TimeNotSynchronized), ("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e", TimeNotSynchronized), ("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620", TimeNotSynchronized), ("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944", TimeNotSynchronized), ("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37", TimeNotSynchronized), ("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396", TimeNotSynchronized), ("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf", TimeNotSynchronized), ("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4", TimeNotSynchronized), ("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 467009 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 26cf176d-2d88-462c-ae8f-5c8d8d132913 response_code = 500 uri = /omicron-zones
28092024-05-08T03:35:36.128ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "26cf176d-2d88-462c-ae8f-5c8d8d132913", "content-length": "124", "date": "Wed, 08 May 2024 03:35:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "26cf176d-2d88-462c-ae8f-5c8d8d132913" } file = sled-agent/src/rack_setup/service.rs:467 retry_after = 35.55166526s sled_address = [fd00:1122:3344:101::1]:12345
28102024-05-08T03:35:36.998ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53597
28112024-05-08T03:35:36.998ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 41 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53597 req_id = 86c9c379-3ea6-489a-b162-6c5c086c9678 response_code = 200 uri = /baseboard
28122024-05-08T03:36:02.847ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 88.483029223s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:426 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
28132024-05-08T03:36:07.036ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56922
28142024-05-08T03:36:07.036ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 29 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56922 req_id = 1c97d88f-a4cc-45af-8161-257445521d2b response_code = 200 uri = /baseboard
28152024-05-08T03:36:11.686ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: dd2b8493-b48e-4834-ad8e-a53d21815a61, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: cc505437-f47e-4e85-84e3-c10ec252eef7, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6151b150-2c83-46b9-8dd9-d454da5235bf, kind: Service { id: 6d4f2f0d-0048-43a0-b797-4120ab6bb8ed }, name: Name("ntp-6d4f2f0d-0048-43a0-b797-4120ab6bb8ed"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 79])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 45e11811-5e91-4520-b80a-7f642e885b6d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: f44a281c-ca8a-47a1-9feb-a5b347543709, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: ec283d93-6725-4976-bed2-c328ce4398e5, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 99ae6502-7f59-4787-8a1e-0923522db0a2, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cfa9067d-2711-42a0-944b-e859db1c0817, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63642991-e779-4349-8697-9bfc6b96d7e4, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: f18a2ae7-5a47-47b8-9eb2-b47a19a2f818, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 44fe13c7-2d3b-4321-9d0e-0f378ee54327, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 109e60dc-df19-44bb-8891-e91ae9e42944, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External } } } }, OmicronZoneConfig { id: d0b8200e-a07a-4794-b87b-17a74dc261e4, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External } } } }, OmicronZoneConfig { id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External } } } }, OmicronZoneConfig { id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External } } } }, OmicronZoneConfig { id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External } } } }, OmicronZoneConfig { id: 46326447-e170-4123-8268-47a9a79585dd, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External } } } }, OmicronZoneConfig { id: 10edf729-c77a-4f8c-9d40-496c39338dcb, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External } } } }, OmicronZoneConfig { id: b8f204fd-3d76-4652-963a-9ee349d4a396, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External } } } }, OmicronZoneConfig { id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:429 sled_address = [fd00:1122:3344:101::1]:12345
28162024-05-08T03:36:11.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28172024-05-08T03:36:11.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dd2b8493-b48e-4834-ad8e-a53d21815a61, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: InternalDns }, responder: ... }
28182024-05-08T03:36:11.707ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28192024-05-08T03:36:11.707ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: cc505437-f47e-4e85-84e3-c10ec252eef7, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: InternalDns }, responder: ... }
28202024-05-08T03:36:11.727ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28212024-05-08T03:36:11.727ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a2adbd8-5cd9-40a0-9c4f-33c9b2f43703, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: InternalDns }, responder: ... }
28222024-05-08T03:36:11.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28232024-05-08T03:36:11.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 45e11811-5e91-4520-b80a-7f642e885b6d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28242024-05-08T03:36:11.765ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28252024-05-08T03:36:11.765ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ae06d4ce-795c-4d42-b79d-9db3f61d73a5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28262024-05-08T03:36:11.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28272024-05-08T03:36:11.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 14c4878d-c5dd-4fc2-8846-aaf3152f4120, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28282024-05-08T03:36:11.800ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28292024-05-08T03:36:11.800ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f44a281c-ca8a-47a1-9feb-a5b347543709, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28302024-05-08T03:36:11.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28312024-05-08T03:36:11.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8a2bf75b-8c7f-4d8d-9ede-6fc064124d25, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28322024-05-08T03:36:11.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28332024-05-08T03:36:11.835ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8b17fb63-5764-4496-bed6-acd4016c4f2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
28342024-05-08T03:36:11.853ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28352024-05-08T03:36:11.853ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ec283d93-6725-4976-bed2-c328ce4398e5, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: ExternalDns }, responder: ... }
28362024-05-08T03:36:11.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
28372024-05-08T03:36:11.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8c94f186-c252-4046-8d01-e8c5cf6a3620, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
28382024-05-08T03:36:11.889ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28392024-05-08T03:36:11.889ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 109e60dc-df19-44bb-8891-e91ae9e42944, dataset_name: DatasetName { pool_name: ZpoolName { id: 58eb44a6-f8c0-46ec-bcea-52bf3efa8996 (zpool), kind: External }, kind: Crucible }, responder: ... }
28402024-05-08T03:36:11.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28412024-05-08T03:36:11.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d0b8200e-a07a-4794-b87b-17a74dc261e4, dataset_name: DatasetName { pool_name: ZpoolName { id: 888a035b-7ecc-4eff-a005-8001c3d3ccff (zpool), kind: External }, kind: Crucible }, responder: ... }
28422024-05-08T03:36:11.924ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28432024-05-08T03:36:11.924ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fe5d8209-53c4-40d5-afba-39343d7ed6c3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1afb2a46-5b47-46d7-9926-14b7e21c6407 (zpool), kind: External }, kind: Crucible }, responder: ... }
28442024-05-08T03:36:11.942ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28452024-05-08T03:36:11.942ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 63dcd254-bbd3-489b-9df5-0db1a21b2d37, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c0211ed-cba6-48d8-8b63-a5f21383d980 (zpool), kind: External }, kind: Crucible }, responder: ... }
28462024-05-08T03:36:11.960ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28472024-05-08T03:36:11.960ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 39fc7bb0-36ee-45b1-96e7-097a2d543d2a, dataset_name: DatasetName { pool_name: ZpoolName { id: fc47eb1d-9a62-49dd-845d-82b29d12e969 (zpool), kind: External }, kind: Crucible }, responder: ... }
28482024-05-08T03:36:11.978ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28492024-05-08T03:36:11.978ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 46326447-e170-4123-8268-47a9a79585dd, dataset_name: DatasetName { pool_name: ZpoolName { id: 62a3be27-a8c4-4992-8657-cb4aacfdb8ab (zpool), kind: External }, kind: Crucible }, responder: ... }
28502024-05-08T03:36:11.996ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28512024-05-08T03:36:11.996ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10edf729-c77a-4f8c-9d40-496c39338dcb, dataset_name: DatasetName { pool_name: ZpoolName { id: 9edc3ed7-0367-4fc4-87ff-888719d0b682 (zpool), kind: External }, kind: Crucible }, responder: ... }
28522024-05-08T03:36:12.016ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28532024-05-08T03:36:12.017ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b8f204fd-3d76-4652-963a-9ee349d4a396, dataset_name: DatasetName { pool_name: ZpoolName { id: ed53811e-9a91-4270-bd15-bcb96ef98de1 (zpool), kind: External }, kind: Crucible }, responder: ... }
28542024-05-08T03:36:12.036ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28552024-05-08T03:36:12.036ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f05d09e0-db61-4cdb-9dbd-1b8480c392bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621 (zpool), kind: External }, kind: Crucible }, responder: ... }
28562024-05-08T03:36:12.053ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28572024-05-08T03:36:12.053ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
28582024-05-08T03:36:12.053ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
28592024-05-08T03:36:12.054ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28602024-05-08T03:36:12.138ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28612024-05-08T03:36:12.151ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28622024-05-08T03:36:12.152ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28632024-05-08T03:36:12.164ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28642024-05-08T03:36:12.165ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28652024-05-08T03:36:12.176ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28662024-05-08T03:36:12.218ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28672024-05-08T03:36:12.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28682024-05-08T03:36:12.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28692024-05-08T03:36:12.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28702024-05-08T03:36:12.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28712024-05-08T03:36:12.365ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
28722024-05-08T03:36:12.365ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28732024-05-08T03:36:12.365ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28742024-05-08T03:36:12.366ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28752024-05-08T03:36:12.400ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28762024-05-08T03:36:12.400ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28772024-05-08T03:36:12.400ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28782024-05-08T03:36:12.400ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28792024-05-08T03:36:12.400ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4 file = illumos-utils/src/zone.rs:315
28802024-05-08T03:36:12.407ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4 file = illumos-utils/src/zone.rs:315
28812024-05-08T03:36:12.412ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620 file = illumos-utils/src/zone.rs:315
28822024-05-08T03:36:12.418ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818 file = illumos-utils/src/zone.rs:315
28832024-05-08T03:36:12.423ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37 file = illumos-utils/src/zone.rs:315
28842024-05-08T03:36:12.427ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3 file = illumos-utils/src/zone.rs:315
28852024-05-08T03:36:12.432ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_46326447-e170-4123-8268-47a9a79585dd file = illumos-utils/src/zone.rs:315
28862024-05-08T03:36:12.568ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4 file = illumos-utils/src/zone.rs:354
28872024-05-08T03:36:12.573ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4 file = illumos-utils/src/zone.rs:354
28882024-05-08T03:36:12.579ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620 file = illumos-utils/src/zone.rs:354
28892024-05-08T03:36:12.584ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818 file = illumos-utils/src/zone.rs:354
28902024-05-08T03:36:12.588ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37 file = illumos-utils/src/zone.rs:354
28912024-05-08T03:36:12.593ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3 file = illumos-utils/src/zone.rs:354
28922024-05-08T03:36:12.597ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_46326447-e170-4123-8268-47a9a79585dd file = illumos-utils/src/zone.rs:354
28932024-05-08T03:36:12.602ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf file = illumos-utils/src/zone.rs:315
28942024-05-08T03:36:12.607ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a file = illumos-utils/src/zone.rs:315
28952024-05-08T03:36:12.613ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e file = illumos-utils/src/zone.rs:315
28962024-05-08T03:36:12.618ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327 file = illumos-utils/src/zone.rs:315
28972024-05-08T03:36:12.624ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb file = illumos-utils/src/zone.rs:315
28982024-05-08T03:36:12.629ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944 file = illumos-utils/src/zone.rs:315
28992024-05-08T03:36:12.635ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396 file = illumos-utils/src/zone.rs:315
29002024-05-08T03:36:12.642ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf file = illumos-utils/src/zone.rs:354
29012024-05-08T03:36:12.647ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a file = illumos-utils/src/zone.rs:354
29022024-05-08T03:36:12.653ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e file = illumos-utils/src/zone.rs:354
29032024-05-08T03:36:12.659ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327 file = illumos-utils/src/zone.rs:354
29042024-05-08T03:36:12.665ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb file = illumos-utils/src/zone.rs:354
29052024-05-08T03:36:12.671ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944 file = illumos-utils/src/zone.rs:354
29062024-05-08T03:36:12.677ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396 file = illumos-utils/src/zone.rs:354
29072024-05-08T03:36:12.736ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
29082024-05-08T03:36:12.809ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
29092024-05-08T03:36:12.875ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
29102024-05-08T03:36:12.935ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
29112024-05-08T03:36:12.997ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
29122024-05-08T03:36:12.998ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29132024-05-08T03:36:12.998ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29142024-05-08T03:36:13.173ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 187, 228]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte1" } }
29152024-05-08T03:36:13.174ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
29162024-05-08T03:36:13.178ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29172024-05-08T03:36:13.178ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29182024-05-08T03:36:13.317ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte2", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 155, 115]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte2" } }
29192024-05-08T03:36:13.317ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = external_dns
29202024-05-08T03:36:13.321ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29212024-05-08T03:36:13.321ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29222024-05-08T03:36:13.436ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte3", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 188, 158]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte3" } }
29232024-05-08T03:36:13.436ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
29242024-05-08T03:36:13.438ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29252024-05-08T03:36:13.438ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29262024-05-08T03:36:13.561ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 160, 210]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte4" } }
29272024-05-08T03:36:13.561ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = external_dns
29282024-05-08T03:36:13.566ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
29292024-05-08T03:36:13.566ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
29302024-05-08T03:36:14.170ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte5", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 170, 42]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte5" } }
29312024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
29322024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
29332024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
29342024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
29352024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
29362024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
29372024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
29382024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
29392024-05-08T03:36:14.170ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
29402024-05-08T03:36:14.766ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
29412024-05-08T03:36:14.848ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
29422024-05-08T03:36:14.858ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
29432024-05-08T03:36:14.859ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
29442024-05-08T03:36:14.859ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
29452024-05-08T03:36:14.859ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29462024-05-08T03:36:14.859ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29472024-05-08T03:36:14.864ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
29482024-05-08T03:36:14.868ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29492024-05-08T03:36:14.868ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29502024-05-08T03:36:14.982ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
29512024-05-08T03:36:14.995ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29522024-05-08T03:36:15.170ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2 file = illumos-utils/src/zone.rs:315
29532024-05-08T03:36:15.298ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d file = illumos-utils/src/zone.rs:315
29542024-05-08T03:36:15.805ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc file = illumos-utils/src/zone.rs:315
29552024-05-08T03:36:15.812ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5 file = illumos-utils/src/zone.rs:315
29562024-05-08T03:36:15.820ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2 file = illumos-utils/src/zone.rs:354
29572024-05-08T03:36:15.834ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d file = illumos-utils/src/zone.rs:354
29582024-05-08T03:36:15.898ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817 file = illumos-utils/src/zone.rs:315
29592024-05-08T03:36:15.908ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc file = illumos-utils/src/zone.rs:354
29602024-05-08T03:36:15.917ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5 file = illumos-utils/src/zone.rs:354
29612024-05-08T03:36:15.944ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817 file = illumos-utils/src/zone.rs:354
29622024-05-08T03:36:28.739ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::f"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29632024-05-08T03:36:28.750ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::10"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29642024-05-08T03:36:28.850ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
29652024-05-08T03:36:28.979ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
29662024-05-08T03:36:29.126ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::d"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value="8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e"/> <propval type="astring" name="address" value="[fd00:1122:3344:101::d]:12223"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29672024-05-08T03:36:29.127ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::11"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29682024-05-08T03:36:29.127ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
29692024-05-08T03:36:29.133ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
29702024-05-08T03:36:29.177ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::15"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_7c0211ed-cba6-48d8-8b63-a5f21383d980/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="63cf92a5-71f5-4106-b58f-8a1fed00b68a"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29712024-05-08T03:36:29.177ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
29722024-05-08T03:36:29.310ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::19"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_ed53811e-9a91-4270-bd15-bcb96ef98de1/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="2c90c7a9-969d-4f3e-9c26-ef0320a1186d"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29732024-05-08T03:36:29.311ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
29742024-05-08T03:36:29.412ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::18"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_9edc3ed7-0367-4fc4-87ff-888719d0b682/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="85419633-865f-4963-9398-7fbec45109b7"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29752024-05-08T03:36:29.412ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
29762024-05-08T03:36:29.428ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::16"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_fc47eb1d-9a62-49dd-845d-82b29d12e969/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="8a213071-d676-41be-987c-b55d40f48e12"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29772024-05-08T03:36:29.428ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
29782024-05-08T03:36:29.436ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::14"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_1afb2a46-5b47-46d7-9926-14b7e21c6407/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="02161613-8e8f-4a3c-8c5f-a88a4de369f7"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29792024-05-08T03:36:29.436ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
29802024-05-08T03:36:29.562ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::13"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_888a035b-7ecc-4eff-a005-8001c3d3ccff/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="0ce16605-2be5-4ca3-9d21-3f07c28a3632"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29812024-05-08T03:36:29.562ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
29822024-05-08T03:36:29.635ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="24d3ed9a-f68f-4424-a293-4492e56df723"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29832024-05-08T03:36:29.637ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService25"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::8"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte4"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::8]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.5:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29842024-05-08T03:36:29.637ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
29852024-05-08T03:36:29.643ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
29862024-05-08T03:36:30.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 26.240249ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
29872024-05-08T03:36:30.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 53.178892ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
29882024-05-08T03:36:30.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 43.116266ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
29892024-05-08T03:36:30.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 68.162002ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
29902024-05-08T03:36:30.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 39.732598ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
29912024-05-08T03:36:30.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 28.553733ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
29922024-05-08T03:36:30.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 72.727406ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
29932024-05-08T03:36:30.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 45.708047ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
29942024-05-08T03:36:30.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 26.595939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
29952024-05-08T03:36:30.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 70.632796ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
29962024-05-08T03:36:30.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 51.286465ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
29972024-05-08T03:36:30.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 65.580436ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
29982024-05-08T03:36:30.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 52.053555ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
29992024-05-08T03:36:30.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 140.875091ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
30002024-05-08T03:36:30.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 98.386162ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
30012024-05-08T03:36:30.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 51.058399ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
30022024-05-08T03:36:30.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 66.070276ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
30032024-05-08T03:36:30.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 139.906214ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
30042024-05-08T03:36:30.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 114.165517ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
30052024-05-08T03:36:30.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 88.564371ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
30062024-05-08T03:36:30.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 120.780191ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
30072024-05-08T03:36:30.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 271.261605ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
30082024-05-08T03:36:30.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 103.662769ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
30092024-05-08T03:36:30.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 126.685867ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
30102024-05-08T03:36:30.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 133.825562ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
30112024-05-08T03:36:30.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 236.674053ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
30122024-05-08T03:36:30.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 180.551232ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
30132024-05-08T03:36:31.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 240.221336ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
30142024-05-08T03:36:31.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 296.129404ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
30152024-05-08T03:36:31.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 268.010098ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
30162024-05-08T03:36:31.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 191.80316ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
30172024-05-08T03:36:31.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 170.690819ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
30182024-05-08T03:36:31.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 193.807224ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
30192024-05-08T03:36:31.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 212.611633ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
30202024-05-08T03:36:31.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 297.823322ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
30212024-05-08T03:36:31.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 295.772214ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
30222024-05-08T03:36:31.141ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService27"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::9"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte2"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::9]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.6:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30232024-05-08T03:36:31.141ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
30242024-05-08T03:36:31.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 514.668904ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
30252024-05-08T03:36:31.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 391.023869ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
30262024-05-08T03:36:31.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 308.580609ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
30272024-05-08T03:36:31.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 363.840751ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
30282024-05-08T03:36:31.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 565.138149ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
30292024-05-08T03:36:31.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 353.652566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
30302024-05-08T03:36:31.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 512.251693ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
30312024-05-08T03:36:31.308ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 385.347871ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
30322024-05-08T03:36:31.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 524.311605ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
30332024-05-08T03:36:31.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 56.197743ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
30342024-05-08T03:36:31.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 385.50345ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
30352024-05-08T03:36:32.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 70.675487ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
30362024-05-08T03:36:32.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 432.461176ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
30372024-05-08T03:36:32.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 748.945591ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
30382024-05-08T03:36:32.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 525.715027ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
30392024-05-08T03:36:32.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 1.09426451s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
30402024-05-08T03:36:32.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 759.422099ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
30412024-05-08T03:36:32.217ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_46326447-e170-4123-8268-47a9a79585dd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::17"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_62a3be27-a8c4-4992-8657-cb4aacfdb8ab/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="35b35f40-bc83-41bc-8d6e-5c7a419a6005"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30422024-05-08T03:36:32.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 1.014910202s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
30432024-05-08T03:36:32.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 593.508026ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
30442024-05-08T03:36:32.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 366.322092ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
30452024-05-08T03:36:32.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 436.759509ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
30462024-05-08T03:36:32.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 1.116294607s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
30472024-05-08T03:36:32.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 536.102844ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
30482024-05-08T03:36:32.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 460.781532ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
30492024-05-08T03:36:32.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 278.794576ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
30502024-05-08T03:36:32.421ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
30512024-05-08T03:36:32.508ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService28"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::c"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte5"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30522024-05-08T03:36:32.508ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
30532024-05-08T03:36:32.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 674.081642ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
30542024-05-08T03:36:32.673ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 57.62696ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
30552024-05-08T03:36:32.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 760.426357ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
30562024-05-08T03:36:32.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.202995759s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
30572024-05-08T03:36:32.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 418.369256ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
30582024-05-08T03:36:32.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 132.533239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
30592024-05-08T03:36:32.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 862.612522ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
30602024-05-08T03:36:32.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 71.709752ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
30612024-05-08T03:36:32.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.409299529s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
30622024-05-08T03:36:32.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 1.157666787s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
30632024-05-08T03:36:32.901ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 202.391966ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
30642024-05-08T03:36:32.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.246317588s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
30652024-05-08T03:36:32.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 51.740622ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
30662024-05-08T03:36:32.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 805.707717ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
30672024-05-08T03:36:32.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 1.350168274s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
30682024-05-08T03:36:33.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 209.431341ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
30692024-05-08T03:36:33.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 584.422222ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
30702024-05-08T03:36:33.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 401.838851ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
30712024-05-08T03:36:33.222ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.017835728s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
30722024-05-08T03:36:33.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 432.558266ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
30732024-05-08T03:36:33.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 592.473551ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
30742024-05-08T03:36:33.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 935.46307ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
30752024-05-08T03:36:33.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 1.316757249s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
30762024-05-08T03:36:33.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 1.331799911s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
30772024-05-08T03:36:33.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.004186832s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
30782024-05-08T03:36:33.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 1.187379892s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
30792024-05-08T03:36:33.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 985.258947ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
30802024-05-08T03:36:33.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 869.649287ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
30812024-05-08T03:36:33.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 1.390138815s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
30822024-05-08T03:36:33.807ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService26"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte1"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30832024-05-08T03:36:33.808ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
30842024-05-08T03:36:33.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 870.38097ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
30852024-05-08T03:36:33.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 611.967779ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
30862024-05-08T03:36:33.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 63.697335ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
30872024-05-08T03:36:34.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 81.861876ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
30882024-05-08T03:36:34.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 981.835161ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
30892024-05-08T03:36:34.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 219.480298ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
30902024-05-08T03:36:34.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.220302097s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
30912024-05-08T03:36:34.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 858.951561ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
30922024-05-08T03:36:34.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 563.115351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
30932024-05-08T03:36:34.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 591.5624ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
30942024-05-08T03:36:34.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 991.523416ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
30952024-05-08T03:36:34.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 421.294221ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
30962024-05-08T03:36:34.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.454022306s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
30972024-05-08T03:36:34.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 945.183939ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
30982024-05-08T03:36:34.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 922.584323ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
30992024-05-08T03:36:34.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 1.34250957s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
31002024-05-08T03:36:34.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 522.450743ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
31012024-05-08T03:36:34.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 836.655449ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
31022024-05-08T03:36:34.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 677.400417ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
31032024-05-08T03:36:34.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 786.535491ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
31042024-05-08T03:36:34.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 911.250188ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
31052024-05-08T03:36:34.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 819.533945ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
31062024-05-08T03:36:34.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.486940988s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
31072024-05-08T03:36:35.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 1.02363121s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
31082024-05-08T03:36:35.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 932.708439ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
31092024-05-08T03:36:35.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 977.928125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
31102024-05-08T03:36:35.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 1.278813496s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
31112024-05-08T03:36:35.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 1.035899401s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
31122024-05-08T03:36:35.398ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.090723902s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
31132024-05-08T03:36:35.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 932.627272ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
31142024-05-08T03:36:35.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 1.407729447s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
31152024-05-08T03:36:35.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 564.016232ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
31162024-05-08T03:36:35.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.32885421s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
31172024-05-08T03:36:35.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 1.085305425s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
31182024-05-08T03:36:35.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 774.25541ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
31192024-05-08T03:36:35.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 870.337598ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
31202024-05-08T03:36:36.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.442521455s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
31212024-05-08T03:36:36.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 697.115127ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
31222024-05-08T03:36:36.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 1.32435415s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
31232024-05-08T03:36:36.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 1.091314369s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
31242024-05-08T03:36:36.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.005245341s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
31252024-05-08T03:36:36.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 998.760893ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
31262024-05-08T03:36:36.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.072506703s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
31272024-05-08T03:36:36.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 600.045071ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
31282024-05-08T03:36:36.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 1.059514746s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
31292024-05-08T03:36:36.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 1.393589087s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
31302024-05-08T03:36:36.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 569.956572ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
31312024-05-08T03:36:36.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 583.808565ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
31322024-05-08T03:36:36.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 760.41278ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
31332024-05-08T03:36:36.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 1.136559917s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
31342024-05-08T03:36:36.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 1.372941608s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
31352024-05-08T03:36:36.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 1.360479473s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
31362024-05-08T03:36:36.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.256910208s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
31372024-05-08T03:36:37.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 1.175175661s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
31382024-05-08T03:36:37.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.130376017s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
31392024-05-08T03:36:37.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 889.515424ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
31402024-05-08T03:36:37.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 884.510187ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
31412024-05-08T03:36:37.206ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 787.979023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
31422024-05-08T03:36:37.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 841.40671ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
31432024-05-08T03:36:37.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 672.672599ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
31442024-05-08T03:36:37.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 965.581038ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
31452024-05-08T03:36:37.405ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46549
31462024-05-08T03:36:37.429ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1861 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46549 req_id = 8546b114-bb36-4e6b-a212-3ff28b886101 response_code = 200 uri = /baseboard
31472024-05-08T03:36:37.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.220783509s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
31482024-05-08T03:36:37.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 806.754208ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
31492024-05-08T03:36:37.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 1.168678304s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
31502024-05-08T03:36:37.789ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::12"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_58eb44a6-f8c0-46ec-bcea-52bf3efa8996/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="2303f7cf-ce9b-4ad4-a723-e7f73bf3bcb3"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31512024-05-08T03:36:37.793ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
31522024-05-08T03:36:37.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 990.629941ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
31532024-05-08T03:36:37.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 1.250761585s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
31542024-05-08T03:36:37.972ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 38.880335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
31552024-05-08T03:36:38.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 1.061361909s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
31562024-05-08T03:36:38.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.003308593s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
31572024-05-08T03:36:38.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 137.129711ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
31582024-05-08T03:36:38.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 511.666489ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
31592024-05-08T03:36:38.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 930.383148ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
31602024-05-08T03:36:38.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 1.140352923s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
31612024-05-08T03:36:38.109ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService24"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte3"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31622024-05-08T03:36:38.109ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
31632024-05-08T03:36:38.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 976.918499ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
31642024-05-08T03:36:38.182ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 295.611396ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
31652024-05-08T03:36:38.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 966.245255ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
31662024-05-08T03:36:38.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 781.242864ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
31672024-05-08T03:36:38.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.160434586s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
31682024-05-08T03:36:38.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 36.989995ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
31692024-05-08T03:36:38.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.465646513s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
31702024-05-08T03:36:38.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 576.330665ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
31712024-05-08T03:36:38.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 85.21141ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
31722024-05-08T03:36:38.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 984.670932ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
31732024-05-08T03:36:38.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 186.133398ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
31742024-05-08T03:36:38.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 379.086628ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
31752024-05-08T03:36:38.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 696.943741ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
31762024-05-08T03:36:38.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 311.478462ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
31772024-05-08T03:36:38.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 906.663329ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
31782024-05-08T03:36:38.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.096136084s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
31792024-05-08T03:36:38.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 953.047876ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
31802024-05-08T03:36:38.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 538.530553ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
31812024-05-08T03:36:38.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 818.360613ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
31822024-05-08T03:36:38.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 623.688366ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
31832024-05-08T03:36:39.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 709.340115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
31842024-05-08T03:36:39.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.283749936s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
31852024-05-08T03:36:39.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.112646585s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
31862024-05-08T03:36:39.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 1.174342195s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
31872024-05-08T03:36:39.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 1.207282355s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
31882024-05-08T03:36:39.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 854.280268ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
31892024-05-08T03:36:39.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 761.403829ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
31902024-05-08T03:36:39.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 1.435938587s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
31912024-05-08T03:36:39.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 731.722939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
31922024-05-08T03:36:39.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 1.134867901s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
31932024-05-08T03:36:39.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.143028717s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
31942024-05-08T03:36:39.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 667.300137ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
31952024-05-08T03:36:39.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 714.431991ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
31962024-05-08T03:36:39.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 641.147554ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
31972024-05-08T03:36:39.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 946.207023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
31982024-05-08T03:36:39.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 739.027812ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
31992024-05-08T03:36:39.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.364372887s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
32002024-05-08T03:36:39.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.004372271s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
32012024-05-08T03:36:39.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 1.26253919s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
32022024-05-08T03:36:39.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 597.404688ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
32032024-05-08T03:36:40.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 866.436004ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
32042024-05-08T03:36:40.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 529.465938ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
32052024-05-08T03:36:40.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 1.147181515s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
32062024-05-08T03:36:40.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.079109044s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
32072024-05-08T03:36:40.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 1.416402814s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
32082024-05-08T03:36:40.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 817.233016ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
32092024-05-08T03:36:40.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 681.377766ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
32102024-05-08T03:36:40.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 1.454084917s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
32112024-05-08T03:36:40.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.0025561s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
32122024-05-08T03:36:40.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 1.006069844s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
32132024-05-08T03:36:40.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 1.429909751s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
32142024-05-08T03:36:40.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 1.160791991s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
32152024-05-08T03:36:40.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 503.536819ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
32162024-05-08T03:36:40.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 1.019851772s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
32172024-05-08T03:36:40.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 503.084836ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
32182024-05-08T03:36:40.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 1.383055051s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
32192024-05-08T03:36:40.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 871.966096ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
32202024-05-08T03:36:40.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 728.53225ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
32212024-05-08T03:36:41.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 947.704321ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
32222024-05-08T03:36:41.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 554.887447ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
32232024-05-08T03:36:41.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 903.978767ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
32242024-05-08T03:36:41.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 872.459744ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
32252024-05-08T03:36:41.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 566.138551ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
32262024-05-08T03:36:41.164ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::e"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
32272024-05-08T03:36:41.168ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32282024-05-08T03:36:41.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 929.832504ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
32292024-05-08T03:36:41.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 695.757749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
32302024-05-08T03:36:41.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 524.05309ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
32312024-05-08T03:36:41.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.393926623s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
32322024-05-08T03:36:41.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 26.300056ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32332024-05-08T03:36:41.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 81.901266ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32342024-05-08T03:36:41.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 1.260597851s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
32352024-05-08T03:36:41.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 260.22775ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32362024-05-08T03:36:41.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 874.678344ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
32372024-05-08T03:36:41.635ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 909.126333ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
32382024-05-08T03:36:41.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.073554314s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
32392024-05-08T03:36:41.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.389477876s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
32402024-05-08T03:36:41.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 914.414902ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
32412024-05-08T03:36:41.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.013798165s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
32422024-05-08T03:36:41.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 502.904241ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
32432024-05-08T03:36:41.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 931.396711ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
32442024-05-08T03:36:41.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 522.203326ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
32452024-05-08T03:36:41.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 498.473362ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32462024-05-08T03:36:41.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 859.93558ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
32472024-05-08T03:36:41.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 808.824768ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
32482024-05-08T03:36:41.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 1.292153562s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
32492024-05-08T03:36:42.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 878.292797ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
32502024-05-08T03:36:42.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 865.139125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
32512024-05-08T03:36:42.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 818.942349ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
32522024-05-08T03:36:42.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 798.109296ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
32532024-05-08T03:36:42.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 778.127007ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
32542024-05-08T03:36:42.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 548.119653ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
32552024-05-08T03:36:42.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 633.03023ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32562024-05-08T03:36:42.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 601.26314ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
32572024-05-08T03:36:42.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 854.738253ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
32582024-05-08T03:36:42.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 727.340194ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
32592024-05-08T03:36:42.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.392275024s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
32602024-05-08T03:36:42.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 790.826521ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
32612024-05-08T03:36:42.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.455894202s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
32622024-05-08T03:36:42.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 619.443032ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
32632024-05-08T03:36:42.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 516.254047ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
32642024-05-08T03:36:42.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 877.475311ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
32652024-05-08T03:36:42.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 1.354529619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
32662024-05-08T03:36:42.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 845.988786ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
32672024-05-08T03:36:42.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 1.279264423s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
32682024-05-08T03:36:42.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.0307711s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
32692024-05-08T03:36:42.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 873.82882ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
32702024-05-08T03:36:42.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 1.119164029s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
32712024-05-08T03:36:43.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 896.113372ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32722024-05-08T03:36:43.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 1.174300011s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
32732024-05-08T03:36:43.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.43665735s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
32742024-05-08T03:36:43.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 1.24371234s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
32752024-05-08T03:36:43.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 866.607447ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
32762024-05-08T03:36:43.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 847.275711ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
32772024-05-08T03:36:43.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 1.358061845s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
32782024-05-08T03:36:43.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 1.108260847s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
32792024-05-08T03:36:43.469ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 1.387662025s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
32802024-05-08T03:36:43.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 1.272198468s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
32812024-05-08T03:36:43.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 1.401435599s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
32822024-05-08T03:36:43.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 987.620011ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
32832024-05-08T03:36:43.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 907.541331ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
32842024-05-08T03:36:43.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 718.290359ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32852024-05-08T03:36:43.983ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.078177464s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
32862024-05-08T03:36:44.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 1.213766771s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
32872024-05-08T03:36:44.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 568.529165ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
32882024-05-08T03:36:44.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 1.240295638s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
32892024-05-08T03:36:44.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.351133944s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
32902024-05-08T03:36:44.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 759.380099ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
32912024-05-08T03:36:44.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 850.799733ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
32922024-05-08T03:36:44.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 986.590185ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
32932024-05-08T03:36:44.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 702.962627ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
32942024-05-08T03:36:44.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 1.43787657s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
32952024-05-08T03:36:44.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.355719694s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
32962024-05-08T03:36:44.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 899.921028ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
32972024-05-08T03:36:44.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 1.358035489s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
32982024-05-08T03:36:44.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.017660914s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
32992024-05-08T03:36:44.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 730.594167ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
33002024-05-08T03:36:44.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 667.214207ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
33012024-05-08T03:36:44.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 1.263265213s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
33022024-05-08T03:36:44.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 1.304046997s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
33032024-05-08T03:36:44.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 1.382697271s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
33042024-05-08T03:36:44.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 1.1062398s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
33052024-05-08T03:36:44.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 908.959142ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
33062024-05-08T03:36:45.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 745.690477ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
33072024-05-08T03:36:45.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 995.878705ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
33082024-05-08T03:36:45.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 779.541286ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
33092024-05-08T03:36:45.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 678.878929ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
33102024-05-08T03:36:45.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 769.094775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
33112024-05-08T03:36:45.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 635.869967ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
33122024-05-08T03:36:45.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 1.227076331s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
33132024-05-08T03:36:45.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 1.361917316s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
33142024-05-08T03:36:45.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 1.208041347s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
33152024-05-08T03:36:45.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 722.063906ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
33162024-05-08T03:36:45.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.425653647s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
33172024-05-08T03:36:45.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 909.30645ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
33182024-05-08T03:36:45.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 1.254182571s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
33192024-05-08T03:36:45.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.069783325s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
33202024-05-08T03:36:45.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 1.06602795s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
33212024-05-08T03:36:45.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 1.491932773s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
33222024-05-08T03:36:45.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 1.05583897s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
33232024-05-08T03:36:46.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 908.726138ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
33242024-05-08T03:36:46.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 560.695728ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
33252024-05-08T03:36:46.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 1.020482719s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
33262024-05-08T03:36:46.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 746.598923ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
33272024-05-08T03:36:46.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 982.633216ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
33282024-05-08T03:36:46.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 685.33686ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
33292024-05-08T03:36:46.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 525.012727ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
33302024-05-08T03:36:46.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 624.482847ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
33312024-05-08T03:36:46.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.173347477s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
33322024-05-08T03:36:46.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 594.503224ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
33332024-05-08T03:36:46.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 1.463003428s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
33342024-05-08T03:36:46.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 698.590338ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
33352024-05-08T03:36:46.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 683.667719ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
33362024-05-08T03:36:46.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.25967529s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
33372024-05-08T03:36:46.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 593.849187ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
33382024-05-08T03:36:46.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 1.133301232s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
33392024-05-08T03:36:46.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 1.359488698s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
33402024-05-08T03:36:46.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e") failed: Property not found. retry in 1.346673307s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
33412024-05-08T03:36:46.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 734.347489ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
33422024-05-08T03:36:47.009ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 548.147666ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
33432024-05-08T03:36:47.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 962.810929ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
33442024-05-08T03:36:47.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 591.058475ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
33452024-05-08T03:36:47.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 1.104022777s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
33462024-05-08T03:36:47.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817") failed: Property not found. retry in 1.417074225s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
33472024-05-08T03:36:47.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327") failed: Property not found. retry in 1.192055669s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
33482024-05-08T03:36:47.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4") failed: Property not found. retry in 1.093242247s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
33492024-05-08T03:36:47.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 638.752254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
33502024-05-08T03:36:47.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2") failed: Property not found. retry in 1.4002371s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
33512024-05-08T03:36:47.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc") failed: Property not found. retry in 1.319618179s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
33522024-05-08T03:36:47.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb") failed: Property not found. retry in 1.107758053s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
33532024-05-08T03:36:47.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 1.3716313s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
33542024-05-08T03:36:47.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a") failed: Property not found. retry in 1.279163075s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
33552024-05-08T03:36:47.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 924.053245ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
33562024-05-08T03:36:47.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 836.777152ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
33572024-05-08T03:36:47.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 514.964433ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
33582024-05-08T03:36:47.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4") failed: Property not found. retry in 1.095517929s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
33592024-05-08T03:36:48.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818") failed: Property not found. retry in 600.8106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
33602024-05-08T03:36:48.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf") failed: Property not found. retry in 1.00205709s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
33612024-05-08T03:36:48.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 1.198211457s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
33622024-05-08T03:36:48.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396") failed: Property not found. retry in 1.076190424s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
33632024-05-08T03:36:48.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37") failed: Property not found. retry in 1.478472614s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
33642024-05-08T03:36:48.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46326447-e170-4123-8268-47a9a79585dd") failed: Property not found. retry in 565.070747ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
33652024-05-08T03:36:48.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 1.448455242s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
33662024-05-08T03:36:48.502ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 522.957071ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
33672024-05-08T03:36:48.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d") failed: Property not found. retry in 1.343813151s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
33682024-05-08T03:36:48.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944") failed: Property not found. retry in 523.418327ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
33692024-05-08T03:36:49.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3") failed: Property not found. retry in 763.85971ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
33702024-05-08T03:36:49.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5") failed: Property not found. retry in 664.940491ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
33712024-05-08T03:36:49.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 621.184926ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
33722024-05-08T03:36:50.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620") failed: Property not found. retry in 830.297071ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
33732024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_4178aff7-3700-4e3f-bdd7-968efe7848fc
33742024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_63642991-e779-4349-8697-9bfc6b96d7e4
33752024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_external_dns_ec283d93-6725-4976-bed2-c328ce4398e5
33762024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_d0b8200e-a07a-4794-b87b-17a74dc261e4
33772024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_99ae6502-7f59-4787-8a1e-0923522db0a2
33782024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_external_dns_8b17fb63-5764-4496-bed6-acd4016c4f2d
33792024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_clickhouse_8c94f186-c252-4046-8d01-e8c5cf6a3620
33802024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_f18a2ae7-5a47-47b8-9eb2-b47a19a2f818
33812024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_63dcd254-bbd3-489b-9df5-0db1a21b2d37
33822024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_fe5d8209-53c4-40d5-afba-39343d7ed6c3
33832024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_46326447-e170-4123-8268-47a9a79585dd
33842024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_f05d09e0-db61-4cdb-9dbd-1b8480c392bf
33852024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_39fc7bb0-36ee-45b1-96e7-097a2d543d2a
33862024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_oximeter_8ab3d3ec-f019-4cb7-bf89-aac78cf3c87e
33872024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_nexus_cfa9067d-2711-42a0-944b-e859db1c0817
33882024-05-08T03:36:51.335ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_pantry_44fe13c7-2d3b-4321-9d0e-0f378ee54327
33892024-05-08T03:36:51.336ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_10edf729-c77a-4f8c-9d40-496c39338dcb
33902024-05-08T03:36:51.336ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_109e60dc-df19-44bb-8891-e91ae9e42944
33912024-05-08T03:36:51.336ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3026 zone = oxz_crucible_b8f204fd-3d76-4652-963a-9ee349d4a396
33922024-05-08T03:36:51.337ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33932024-05-08T03:36:51.340ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33942024-05-08T03:36:51.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 39654677 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64762 req_id = 9008cab9-6326-4907-a1a9-c3fd5c6cde9b response_code = 204 uri = /omicron-zones
33952024-05-08T03:36:51.375ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1212
33962024-05-08T03:36:51.375ZINFOSledAgent (RSS): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
33972024-05-08T03:36:51.376ZINFOSledAgent (RSS): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
33982024-05-08T03:36:51.380ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:679
33992024-05-08T03:36:51.384ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:692
34002024-05-08T03:36:51.450ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV1 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV1 { addresses: [ V4( Ipv4Network { addr: 10.151.1.174, prefix: 24, }, ), ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.1.100, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, } file = sled-agent/src/rack_setup/service.rs:803
34012024-05-08T03:36:51.456ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34022024-05-08T03:36:51.545ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34032024-05-08T03:36:51.685ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34042024-05-08T03:36:51.766ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34052024-05-08T03:36:51.869ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34062024-05-08T03:36:52.067ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34072024-05-08T03:36:52.364ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34082024-05-08T03:36:52.541ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34092024-05-08T03:36:52.852ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34102024-05-08T03:36:53.351ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34112024-05-08T03:36:53.905ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34122024-05-08T03:36:54.222ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34132024-05-08T03:36:54.818ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34142024-05-08T03:36:55.558ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34152024-05-08T03:36:56.657ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34162024-05-08T03:36:58.104ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34172024-05-08T03:36:59.724ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/76e0fa5a-5dcb-4021-ab9c-640c3670ef01/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:855
34182024-05-08T03:37:00.880ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "980e3d9d-7047-4041-88cc-9924138d031d", "content-length": "123", "date": "Wed, 08 May 2024 03:37:00 GMT"}; value: Error { error_code: Some("InvalidRequest"), message: "rack not found", request_id: "980e3d9d-7047-4041-88cc-9924138d031d" } file = sled-agent/src/rack_setup/service.rs:855
34192024-05-08T03:37:02.476ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1b40eba8-2475-4fff-a029-875712106e31", "content-length": "124", "date": "Wed, 08 May 2024 03:37:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1b40eba8-2475-4fff-a029-875712106e31" } file = sled-agent/src/rack_setup/service.rs:855
34202024-05-08T03:37:05.432ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ea6d5b55-ba70-46c2-b186-306625a75b0e", "content-length": "124", "date": "Wed, 08 May 2024 03:37:05 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ea6d5b55-ba70-46c2-b186-306625a75b0e" } file = sled-agent/src/rack_setup/service.rs:855
34212024-05-08T03:37:06.987ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
34222024-05-08T03:37:07.124ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35256
34232024-05-08T03:37:07.126ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34242024-05-08T03:37:07.218ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 93518 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35256 req_id = 3b403a0e-3a56-4c29-89b0-982cb15ec4a2 response_code = 200 uri = /inventory
34252024-05-08T03:37:07.218ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56536
34262024-05-08T03:37:07.219ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 461 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56536 req_id = 2e819326-358f-437a-a7a3-dd982729e34c response_code = 200 uri = /network-bootstore-config
34272024-05-08T03:37:07.219ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34282024-05-08T03:37:07.219ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
34292024-05-08T03:37:07.221ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
34302024-05-08T03:37:07.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3152 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35256 req_id = 26331448-1687-475e-b01e-41a04a7917bc response_code = 200 uri = /omicron-zones
34312024-05-08T03:37:07.475ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49569
34322024-05-08T03:37:07.476ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 58 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:49569 req_id = e63a6344-7edc-49a1-8c42-d72641ac5145 response_code = 200 uri = /baseboard
34332024-05-08T03:37:08.368ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41998
34342024-05-08T03:37:08.372ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6aac3518-7da3-4a4d-8b34-c4cfc1e83da8, kind: Service { id: 8b17fb63-5764-4496-bed6-acd4016c4f2d }, name: Name("external-dns-8b17fb63-5764-4496-bed6-acd4016c4f2d"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 210])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: b7228daa-3ec3-4bd0-8f53-26d1e2f21a08, kind: Service { id: ec283d93-6725-4976-bed2-c328ce4398e5 }, name: Name("external-dns-ec283d93-6725-4976-bed2-c328ce4398e5"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 155, 115])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 48680075-02d3-4e9c-afb5-2cb5e59f4ff4, kind: Service { id: 4178aff7-3700-4e3f-bdd7-968efe7848fc }, name: Name("nexus-4178aff7-3700-4e3f-bdd7-968efe7848fc"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 187, 228])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 57ab3fcc-eed9-42b4-9966-1c2fd43d1479, kind: Service { id: cfa9067d-2711-42a0-944b-e859db1c0817 }, name: Name("nexus-cfa9067d-2711-42a0-944b-e859db1c0817"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 42])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 68a555c7-fa8f-4574-a7ca-75539f197e18, kind: Service { id: 99ae6502-7f59-4787-8a1e-0923522db0a2 }, name: Name("nexus-99ae6502-7f59-4787-8a1e-0923522db0a2"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 158])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34352024-05-08T03:37:08.372ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34362024-05-08T03:37:08.372ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34372024-05-08T03:37:08.372ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
34382024-05-08T03:37:08.372ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34392024-05-08T03:37:08.373ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34402024-05-08T03:37:08.373ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34412024-05-08T03:37:08.373ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 5294 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41998 req_id = e59edf9f-ff5b-408d-be08-119ee4641477 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34422024-05-08T03:37:08.374ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:865
34432024-05-08T03:37:08.700ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64737
34442024-05-08T03:37:08.700ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34452024-05-08T03:37:08.787ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 87276 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64737 req_id = a8e3ddb7-92df-4aed-99c2-f7c2a6cc0e85 response_code = 200 uri = /inventory
34462024-05-08T03:37:08.788ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34472024-05-08T03:37:08.788ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/config/omicron-zones.json file = common/src/ledger.rs:177
34482024-05-08T03:37:08.788ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/config/omicron-zones.json file = common/src/ledger.rs:177
34492024-05-08T03:37:08.789ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 943 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64737 req_id = 5cb3f53b-30da-44fb-9297-cbe97b57681a response_code = 200 uri = /omicron-zones
34502024-05-08T03:37:30.904ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63035
34512024-05-08T03:37:30.986ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 81534 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63035 req_id = 0eb975f8-f708-4df1-b4c1-f8b06450349a response_code = 204 uri = /switch-ports
34522024-05-08T03:37:30.986ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58372
34532024-05-08T03:37:31.039ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 52787 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58372 req_id = e8795b12-56d4-49f3-ae5b-baccf389adf3 response_code = 204 uri = /switch-ports
34542024-05-08T03:37:31.039ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63035 req_id = dad43c2b-4b8a-4f88-8613-93f0492f6343 response_code = 200 uri = /network-bootstore-config
34552024-05-08T03:37:31.039ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58372 req_id = 577793d6-c79f-4a5f-87e2-0854c63524d6 response_code = 200 uri = /network-bootstore-config
34562024-05-08T03:37:31.062ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
34572024-05-08T03:37:31.062ZINFOSledAgent (bootstore): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
34582024-05-08T03:37:31.063ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43680
34592024-05-08T03:37:31.064ZINFOSledAgent (bootstore): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
34602024-05-08T03:37:31.068ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
34612024-05-08T03:37:31.068ZINFOSledAgent (bootstore): Attempting to update network config with generation: 3, current_generation: 2 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
34622024-05-08T03:37:31.068ZINFOSledAgent (bootstore): Writing ledger to /pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
34632024-05-08T03:37:31.068ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 6165 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63035 req_id = bdf0bfc7-a1f1-4bf9-bb5f-e6b5aeb71e2f response_code = 204 uri = /network-bootstore-config
34642024-05-08T03:37:31.068ZINFOSledAgent (bootstore): Writing ledger to /pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
34652024-05-08T03:37:31.068ZINFOSledAgent (bootstore): Broadcasting network config with generation 3 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
34662024-05-08T03:37:31.068ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1240 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58372 req_id = 3570f5ff-f6e8-4be8-8ac6-a06799de9575 response_code = 204 uri = /network-bootstore-config
34672024-05-08T03:37:31.109ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45913 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43680 req_id = 664311f2-40cb-4faf-a147-b6ba00a74d63 response_code = 204 uri = /switch-ports
34682024-05-08T03:37:31.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43680 req_id = 0634f4ea-45e0-41b4-9558-3d151264d625 response_code = 200 uri = /network-bootstore-config
34692024-05-08T03:37:37.510ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45399
34702024-05-08T03:37:37.510ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 119 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45399 req_id = e0298a49-29f4-40e1-968e-f7cea27c10f2 response_code = 200 uri = /baseboard
34712024-05-08T03:37:52.855ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
34722024-05-08T03:38:01.011ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61617
34732024-05-08T03:38:01.056ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44216 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61617 req_id = 5eb40830-a35c-4a35-b109-e9eb2afe6ddb response_code = 204 uri = /switch-ports
34742024-05-08T03:38:01.056ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61617 req_id = c471b5eb-b2e5-4ab0-8ae3-78d170ae42a5 response_code = 200 uri = /network-bootstore-config
34752024-05-08T03:38:01.151ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44336
34762024-05-08T03:38:01.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46148 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44336 req_id = e9ffb556-fdc3-412f-9260-8f2ae3647f08 response_code = 204 uri = /switch-ports
34772024-05-08T03:38:01.197ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44336 req_id = 0a5edbc5-e664-4a3a-82a1-887fe5a59cc4 response_code = 200 uri = /network-bootstore-config
34782024-05-08T03:38:01.372ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39529
34792024-05-08T03:38:01.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47317 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39529 req_id = 90bbc7ed-2bd9-4bf7-8ddf-c2ac939388ef response_code = 204 uri = /switch-ports
34802024-05-08T03:38:01.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39529 req_id = 954b6af9-f374-423f-af0b-2ac3ea2c7f71 response_code = 200 uri = /network-bootstore-config
34812024-05-08T03:38:01.449ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:34994 remote_addr = [fd00:1122:3344:101::d]:47538 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732
34822024-05-08T03:38:01.460ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 9947 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = e01d8253-480d-452b-a891-d9ed98b305d6 response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
34832024-05-08T03:38:07.544ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35751
34842024-05-08T03:38:07.545ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 74 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35751 req_id = c39aed6a-8516-45ea-885a-24c3f7530158 response_code = 200 uri = /baseboard
34852024-05-08T03:38:30.869ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40172
34862024-05-08T03:38:30.919ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 49939 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40172 req_id = 2c1f1cca-90b1-4d0c-a3c1-63bc93f673ab response_code = 204 uri = /switch-ports
34872024-05-08T03:38:30.919ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40527
34882024-05-08T03:38:30.922ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3244 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40172 req_id = db71312d-c157-44aa-9667-287088b42dcd response_code = 200 uri = /network-bootstore-config
34892024-05-08T03:38:30.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47932 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40527 req_id = 22324d9c-2282-41f0-9ba4-ff3493d4c520 response_code = 204 uri = /switch-ports
34902024-05-08T03:38:30.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40527 req_id = b0deddb7-63b8-42bf-92aa-7cdd319cb90f response_code = 200 uri = /network-bootstore-config
34912024-05-08T03:38:31.221ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41023
34922024-05-08T03:38:31.270ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 49417 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41023 req_id = b502e07e-d705-4842-8ca2-7f79708804f6 response_code = 204 uri = /switch-ports
34932024-05-08T03:38:31.270ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41023 req_id = f70909bd-f44b-4ccd-a08f-b5dc825e15d2 response_code = 200 uri = /network-bootstore-config
34942024-05-08T03:38:31.450ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1324 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = 66bb7598-55f8-4191-ad2b-28ab1de13326 response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
34952024-05-08T03:38:37.579ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60903
34962024-05-08T03:38:37.579ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 71 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60903 req_id = ce24b3f5-1f56-41f2-bc24-750dcb9ca402 response_code = 200 uri = /baseboard
34972024-05-08T03:39:00.992ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49636
34982024-05-08T03:39:01.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 50458 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49636 req_id = 8caf4a2a-1f53-46a8-9d3c-48d9e27efacc response_code = 204 uri = /switch-ports
34992024-05-08T03:39:01.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49636 req_id = cf1ac6e7-e5cd-409c-a92a-41fb87b82bd6 response_code = 200 uri = /network-bootstore-config
35002024-05-08T03:39:01.061ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51909
35012024-05-08T03:39:01.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44975 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:51909 req_id = 7ee17b2d-c7b8-4f9b-b30e-e4da1b50d1ed response_code = 204 uri = /switch-ports
35022024-05-08T03:39:01.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51909 req_id = 06ff2ce9-70c8-4865-a99b-7a166013cbc6 response_code = 200 uri = /network-bootstore-config
35032024-05-08T03:39:01.304ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65112
35042024-05-08T03:39:01.352ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47654 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:65112 req_id = 52b37d6b-1d28-4351-91d2-9632ec8802f5 response_code = 204 uri = /switch-ports
35052024-05-08T03:39:01.353ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 694 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65112 req_id = de123faf-d5c6-4136-9e1c-201de98c4ce4 response_code = 200 uri = /network-bootstore-config
35062024-05-08T03:39:01.450ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 799 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = 4ee882be-aefd-41ad-98bb-dca3eddf7ec8 response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
35072024-05-08T03:39:07.613ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48484
35082024-05-08T03:39:07.613ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 71 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48484 req_id = 0e0eb5ec-2c86-420a-8853-14252aa1df73 response_code = 200 uri = /baseboard
35092024-05-08T03:39:30.863ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48520
35102024-05-08T03:39:30.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 53061 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48520 req_id = 97bcc445-18d2-42cb-9984-85f5e1de6df8 response_code = 204 uri = /switch-ports
35112024-05-08T03:39:30.918ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 947 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48520 req_id = 8244fa3e-d793-476f-98e8-25ef52270b7c response_code = 200 uri = /network-bootstore-config
35122024-05-08T03:39:30.918ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64213
35132024-05-08T03:39:30.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45655 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64213 req_id = bb6e3aea-95c7-49a7-b254-ae0a96412781 response_code = 204 uri = /switch-ports
35142024-05-08T03:39:30.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64213 req_id = 2aa09955-63df-41db-b741-4948de0fcac5 response_code = 200 uri = /network-bootstore-config
35152024-05-08T03:39:30.984ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49013
35162024-05-08T03:39:31.032ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 48339 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49013 req_id = 258cb2ad-e2fc-4ae2-a754-145d64e83c6e response_code = 204 uri = /switch-ports
35172024-05-08T03:39:31.032ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49013 req_id = 4e941d8e-8d99-4477-88fc-814ec87e6146 response_code = 200 uri = /network-bootstore-config
35182024-05-08T03:39:31.451ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 714 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = f3e82209-e0e9-4fee-9028-c139f19569c2 response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
35192024-05-08T03:39:37.648ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61436
35202024-05-08T03:39:37.648ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 75 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61436 req_id = 00739959-aec0-4fce-b561-7ff6d9023d5b response_code = 200 uri = /baseboard
35212024-05-08T03:40:00.990ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40924
35222024-05-08T03:40:01.038ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 48245 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:40924 req_id = 00715f35-24c8-49a8-80d1-512e06e53ca2 response_code = 204 uri = /switch-ports
35232024-05-08T03:40:01.038ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40924 req_id = 8a818061-4a9b-46bf-8a58-91835b946d6f response_code = 200 uri = /network-bootstore-config
35242024-05-08T03:40:01.160ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55179
35252024-05-08T03:40:01.206ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46250 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:55179 req_id = e0ba1a61-8cbb-462d-b305-b052b09922ac response_code = 204 uri = /switch-ports
35262024-05-08T03:40:01.206ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55179 req_id = 98e6a134-9add-42d6-89a6-05010bb82146 response_code = 200 uri = /network-bootstore-config
35272024-05-08T03:40:01.239ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63864
35282024-05-08T03:40:01.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43112 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63864 req_id = de1b1777-f36e-41b6-b65b-92f1496a0ad2 response_code = 204 uri = /switch-ports
35292024-05-08T03:40:01.283ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63864 req_id = 310de80e-0f94-4e6d-880d-ca0082fdde49 response_code = 200 uri = /network-bootstore-config
35302024-05-08T03:40:01.470ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 442 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = 11d99394-2265-41de-97c7-8221c3de5e5f response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
35312024-05-08T03:40:07.683ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61748
35322024-05-08T03:40:07.683ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 95 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61748 req_id = c19e9784-6379-4e5a-9c3f-c049682de726 response_code = 200 uri = /baseboard
35332024-05-08T03:40:30.794ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58643
35342024-05-08T03:40:30.842ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47269 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58643 req_id = 98025006-cd24-45f4-940e-88dd172ba330 response_code = 204 uri = /switch-ports
35352024-05-08T03:40:30.842ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58643 req_id = 6e645963-d456-4100-ba1d-49e1fe03949f response_code = 200 uri = /network-bootstore-config
35362024-05-08T03:40:30.940ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35445
35372024-05-08T03:40:30.982ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 42439 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35445 req_id = da84ca75-70c2-4c89-9048-875e405263bf response_code = 204 uri = /switch-ports
35382024-05-08T03:40:30.982ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35445 req_id = e88c37fe-da28-436a-90fb-40b0c12d2fd3 response_code = 200 uri = /network-bootstore-config
35392024-05-08T03:40:31.093ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38675
35402024-05-08T03:40:31.137ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44663 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:38675 req_id = 8dd479a3-f04b-425f-b904-83e2e088bb19 response_code = 204 uri = /switch-ports
35412024-05-08T03:40:31.138ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38675 req_id = 1cf4ed02-2b2a-45ed-b8d1-3a6488de0b6a response_code = 200 uri = /network-bootstore-config
35422024-05-08T03:40:31.515ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 2820 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = c4efb6bc-81d2-4ebb-823e-05b0e8f1053e response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
35432024-05-08T03:40:37.717ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38169
35442024-05-08T03:40:37.717ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 82 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38169 req_id = 8115e16f-cf9f-464f-93c0-4c740cce34f8 response_code = 200 uri = /baseboard
35452024-05-08T03:41:01.048ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33830
35462024-05-08T03:41:01.095ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46699 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33830 req_id = 3e010485-5ff1-487b-901d-b96cdaa72883 response_code = 204 uri = /switch-ports
35472024-05-08T03:41:01.095ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33830 req_id = b8991c53-9e35-44ff-a570-1bf126755cd5 response_code = 200 uri = /network-bootstore-config
35482024-05-08T03:41:01.243ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42545
35492024-05-08T03:41:01.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46856 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:42545 req_id = 3f042de4-ac5d-4dc9-b5b2-73453657f98a response_code = 204 uri = /switch-ports
35502024-05-08T03:41:01.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42545 req_id = a52a4f81-298a-4d98-aaad-61313fb823de response_code = 200 uri = /network-bootstore-config
35512024-05-08T03:41:01.515ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1245 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = 6be7e87f-a9b0-4fcc-8129-f74c36a83393 response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
35522024-05-08T03:41:01.537ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50007
35532024-05-08T03:41:01.584ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47175 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50007 req_id = 9b14b92f-ff20-4650-9f53-db6aae1b84cd response_code = 204 uri = /switch-ports
35542024-05-08T03:41:01.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50007 req_id = 2ebf0059-3cc3-4ff8-b8e0-a397fd2d61d3 response_code = 200 uri = /network-bootstore-config
35552024-05-08T03:41:07.750ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37846
35562024-05-08T03:41:07.750ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 83 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37846 req_id = 08771a7c-60d1-4485-ad98-f68dfa6994ee response_code = 200 uri = /baseboard
35572024-05-08T03:41:30.856ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54991
35582024-05-08T03:41:30.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47991 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54991 req_id = 5d854874-2472-4b04-9f2b-5f7a966b648c response_code = 204 uri = /switch-ports
35592024-05-08T03:41:30.904ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63343
35602024-05-08T03:41:30.908ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3218 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54991 req_id = 0d0261dc-0e69-4363-9990-951430260dd5 response_code = 200 uri = /network-bootstore-config
35612024-05-08T03:41:30.953ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 48315 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63343 req_id = 3835aaef-5d6c-4cf9-80db-5be309713158 response_code = 204 uri = /switch-ports
35622024-05-08T03:41:30.954ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 781 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63343 req_id = d76a462e-7434-488a-a560-198ba86a2585 response_code = 200 uri = /network-bootstore-config
35632024-05-08T03:41:30.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60001
35642024-05-08T03:41:31.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43571 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60001 req_id = f02a26e7-7997-4013-b247-c9c4060d213c response_code = 204 uri = /switch-ports
35652024-05-08T03:41:31.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60001 req_id = 017f0ef0-6120-4e06-b1f0-216ba6baead7 response_code = 200 uri = /network-bootstore-config
35662024-05-08T03:41:31.515ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 557 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = f373956d-5403-41ef-abf4-3b8965d26f0f response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
35672024-05-08T03:41:37.784ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37550
35682024-05-08T03:41:37.784ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 722 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37550 req_id = 7bbf09b4-e6d4-4d3d-8f6d-79445e76fd7f response_code = 200 uri = /baseboard
35692024-05-08T03:42:01.100ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34443
35702024-05-08T03:42:01.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43952 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34443 req_id = ffd3b7ef-7cff-4ac6-9dc1-a9c819a6cd10 response_code = 204 uri = /switch-ports
35712024-05-08T03:42:01.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34443 req_id = a1db61ed-d161-4baf-95e2-4622ce4f6bec response_code = 200 uri = /network-bootstore-config
35722024-05-08T03:42:01.190ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59048
35732024-05-08T03:42:01.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44001 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59048 req_id = 43bcf250-9e54-4fc8-9deb-0236bc471b41 response_code = 204 uri = /switch-ports
35742024-05-08T03:42:01.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59048 req_id = 5a396b15-b4ef-4ab0-9bfc-af0a65838c33 response_code = 200 uri = /network-bootstore-config
35752024-05-08T03:42:01.236ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50888
35762024-05-08T03:42:01.279ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43134 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50888 req_id = 09ba4e82-6162-4b42-8a7a-655fbf1f4cde response_code = 204 uri = /switch-ports
35772024-05-08T03:42:01.280ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 809 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50888 req_id = dfc23745-3baf-46b7-a88b-4dfca32b26ab response_code = 200 uri = /network-bootstore-config
35782024-05-08T03:42:01.515ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 625 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = 872bfea9-8f9a-4be0-816f-6ab5c4a8d7a4 response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
35792024-05-08T03:42:07.819ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47867
35802024-05-08T03:42:07.820ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 747 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47867 req_id = 4e8eeb3a-3e42-4626-813b-3e61969b4d65 response_code = 200 uri = /baseboard
35812024-05-08T03:42:10.933ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:300
35822024-05-08T03:42:10.934ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
35832024-05-08T03:42:10.972ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 11328, tv_nsec: 48069197 } time_to_next_cleanup = 599.99999397s
35842024-05-08T03:42:30.858ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47498
35852024-05-08T03:42:30.905ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46904 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47498 req_id = 6a0b5d86-5a07-4498-a2cc-f5133727e719 response_code = 204 uri = /switch-ports
35862024-05-08T03:42:30.905ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48723
35872024-05-08T03:42:30.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 739 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47498 req_id = 98bfcf3e-d793-4fcb-a0e5-e3c0e5ab4a7b response_code = 200 uri = /network-bootstore-config
35882024-05-08T03:42:30.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45278 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48723 req_id = 213e677f-4869-45c4-a691-c275f31ff38f response_code = 204 uri = /switch-ports
35892024-05-08T03:42:30.951ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 153 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48723 req_id = a20c3774-a4cb-4b25-ab71-25f058aceef1 response_code = 200 uri = /network-bootstore-config
35902024-05-08T03:42:31.034ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33698
35912024-05-08T03:42:31.077ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43183 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:33698 req_id = 6743113f-8e8f-45a2-ba7f-66046f2b0d15 response_code = 204 uri = /switch-ports
35922024-05-08T03:42:31.077ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33698 req_id = c723bfa4-40e1-4616-980a-3702f4be62c9 response_code = 200 uri = /network-bootstore-config
35932024-05-08T03:42:31.515ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 571 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = 161b845f-32a2-454e-a6ad-babe5c57e00a response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
35942024-05-08T03:42:37.853ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57234
35952024-05-08T03:42:37.853ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 93 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57234 req_id = 18035d69-e11a-4b4f-929a-d16cc99741da response_code = 200 uri = /baseboard
35962024-05-08T03:42:52.855ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
35972024-05-08T03:43:01.157ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34738
35982024-05-08T03:43:01.201ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43971 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34738 req_id = 52799a91-fe20-440c-9b85-991fe6cd5c19 response_code = 204 uri = /switch-ports
35992024-05-08T03:43:01.202ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34770
36002024-05-08T03:43:01.202ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 261 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34738 req_id = 90ef636c-bfa6-4a99-b022-ca6c53fa4981 response_code = 200 uri = /network-bootstore-config
36012024-05-08T03:43:01.248ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46573 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34770 req_id = 9a2ba46f-0a00-4a99-8766-39e2e0631423 response_code = 204 uri = /switch-ports
36022024-05-08T03:43:01.249ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34770 req_id = cf9b6adc-19ef-413c-a6c8-f52985ac5642 response_code = 200 uri = /network-bootstore-config
36032024-05-08T03:43:01.261ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52600
36042024-05-08T03:43:01.300ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 39362 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52600 req_id = 3de36649-6428-446e-87b6-644e1ff502a7 response_code = 204 uri = /switch-ports
36052024-05-08T03:43:01.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52600 req_id = 7a72558f-9d45-4b7c-a04e-99ba788127df response_code = 200 uri = /network-bootstore-config
36062024-05-08T03:43:01.645ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 603 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = 66837757-75b4-4eb7-b7a5-3be530b76e70 response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
36072024-05-08T03:43:07.887ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62464
36082024-05-08T03:43:07.887ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 83 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62464 req_id = a0f21680-8619-4a29-9d50-4ad692cf4637 response_code = 200 uri = /baseboard
36092024-05-08T03:43:30.806ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38901
36102024-05-08T03:43:30.857ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 50899 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:38901 req_id = 1ad64204-5757-4afd-a56d-2118b6fea574 response_code = 204 uri = /switch-ports
36112024-05-08T03:43:30.857ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56539
36122024-05-08T03:43:30.857ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 234 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38901 req_id = 8cf19eb5-ef5b-49d4-995e-8825e510c6c8 response_code = 200 uri = /network-bootstore-config
36132024-05-08T03:43:30.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46814 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56539 req_id = c1ab6b13-38dd-4461-98c2-2ff5244e24a2 response_code = 204 uri = /switch-ports
36142024-05-08T03:43:30.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56539 req_id = fd0802e9-7048-46d8-b785-c947258f68c4 response_code = 200 uri = /network-bootstore-config
36152024-05-08T03:43:30.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39539
36162024-05-08T03:43:31.005ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44427 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:39539 req_id = db299c03-92f9-4491-b9b8-a94edf846f8a response_code = 204 uri = /switch-ports
36172024-05-08T03:43:31.005ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39539 req_id = 05866559-be75-40e5-a5dd-61b8ed4741e5 response_code = 200 uri = /network-bootstore-config
36182024-05-08T03:43:31.646ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1623 local_addr = [fd00:1122:3344:101::1]:34994 method = GET remote_addr = [fd00:1122:3344:101::d]:47538 req_id = 56cd355c-0e32-4f5f-b02e-9cd5b48735bf response_code = 200 sled_id = 411336b5-dbaf-48b1-93b3-a95d20f0f732 uri = //411336b5-dbaf-48b1-93b3-a95d20f0f732
36192024-05-08T03:43:32.154ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57706
36202024-05-08T03:43:32.166ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:521 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 3cfb1958-3aef-42f5-bea9-40aca6841824, kind: Instance { id: bda7dea9-9e54-480f-acc5-e5575f71172d }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 249, 215, 108])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(3235028), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3cfb1958-3aef-42f5-bea9-40aca6841824, kind: Instance { id: bda7dea9-9e54-480f-acc5-e5575f71172d }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 249, 215, 108])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(3235028), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3cfb1958-3aef-42f5-bea9-40aca6841824, kind: Instance { id: bda7dea9-9e54-480f-acc5-e5575f71172d }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 249, 215, 108])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(3235028), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(3235028))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3cfb1958-3aef-42f5-bea9-40aca6841824, kind: Instance { id: bda7dea9-9e54-480f-acc5-e5575f71172d }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 249, 215, 108])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(3235028), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-afd9851865e", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: e8cbe33e-b6b0-4ad4-96b6-1544ea31e55b, read_only_parent: Some(Volume { block_size: 512, id: 2728540c-b0df-45e1-941b-f3cc5a445341, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 738959c6-c771-4d96-955e-1d1120abe8a6, key: Some("ROJ+L4dC8b6WoTWPPXqMnTdQiJo6WFiQzGfCLsVVwrk="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: e8cbe33e-b6b0-4ad4-96b6-1544ea31e55b, key: Some("ENl72cpprsNbdNCXbNY1JquTakKEw0CETtyguYvYHtY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::16]:19002", "[fd00:1122:3344:101::15]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAALcB2oWKhYAABwHahYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAC3AdqFioWAAAcB2oWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJiZGE3ZGVhOS05ZTU0LTQ4MGYtYWNjNS1lNTU3NWY3MTE3MmQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRExTY1lsdVAvd2RCVCs3L0FJK1M5MW9UbU95WFpTOUVkODJyTDErclhobCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d instance_runtime = InstanceRuntimeState { propolis_id: Some(ef94a682-3038-4280-bef1-5a995ac28fbd), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-08T03:43:31.727598140Z } metadata = InstanceMetadata { silo_id: 8dac2178-2bd9-4d3d-ae2d-15851d0211c3, project_id: 6120e812-bc0d-472f-9b7c-f86cb8218397 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = ef94a682-3038-4280-bef1-5a995ac28fbd vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-05-08T03:43:31.700635Z }
36212024-05-08T03:43:32.166ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:555 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36222024-05-08T03:43:32.166ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:938 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d propolis_id = ef94a682-3038-4280-bef1-5a995ac28fbd state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 3cfb1958-3aef-42f5-bea9-40aca6841824, kind: Instance { id: bda7dea9-9e54-480f-acc5-e5575f71172d }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 249, 215, 108])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(3235028), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3cfb1958-3aef-42f5-bea9-40aca6841824, kind: Instance { id: bda7dea9-9e54-480f-acc5-e5575f71172d }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 249, 215, 108])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(3235028), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3cfb1958-3aef-42f5-bea9-40aca6841824, kind: Instance { id: bda7dea9-9e54-480f-acc5-e5575f71172d }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 249, 215, 108])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(3235028), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(3235028))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 3cfb1958-3aef-42f5-bea9-40aca6841824, kind: Instance { id: bda7dea9-9e54-480f-acc5-e5575f71172d }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 249, 215, 108])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(3235028), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-afd9851865e", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: e8cbe33e-b6b0-4ad4-96b6-1544ea31e55b, read_only_parent: Some(Volume { block_size: 512, id: 2728540c-b0df-45e1-941b-f3cc5a445341, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 738959c6-c771-4d96-955e-1d1120abe8a6, key: Some("ROJ+L4dC8b6WoTWPPXqMnTdQiJo6WFiQzGfCLsVVwrk="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: e8cbe33e-b6b0-4ad4-96b6-1544ea31e55b, key: Some("ENl72cpprsNbdNCXbNY1JquTakKEw0CETtyguYvYHtY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::16]:19002", "[fd00:1122:3344:101::15]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAALcB2oWKhYAABwHahYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAC3AdqFioWAAAcB2oWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJiZGE3ZGVhOS05ZTU0LTQ4MGYtYWNjNS1lNTU3NWY3MTE3MmQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRExTY1lsdVAvd2RCVCs3L0FJK1M5MW9UbU95WFpTOUVkODJyTDErclhobCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(ef94a682-3038-4280-bef1-5a995ac28fbd), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-08T03:43:31.727598140Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-05-08T03:43:31.700635Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
36232024-05-08T03:43:32.167ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 12344 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57706 req_id = 9994b976-2407-46df-9779-7cbf4f5a214f response_code = 200 uri = /instances/bda7dea9-9e54-480f-acc5-e5575f71172d
36242024-05-08T03:43:32.282ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35581
36252024-05-08T03:43:32.283ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1413 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:35581 req_id = d284c329-5af2-4c71-98c6-7ea48a5265fd response_code = 200 uri = /instances/bda7dea9-9e54-480f-acc5-e5575f71172d/state
36262024-05-08T03:43:32.303ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 249, 215, 108]), slot: 0, vni: Vni { inner: 3235028 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 }, vnic: "vopte6" } }
36272024-05-08T03:43:32.303ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
36282024-05-08T03:43:32.338ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd file = illumos-utils/src/zone.rs:315 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36292024-05-08T03:43:32.359ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd file = illumos-utils/src/zone.rs:354 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36302024-05-08T03:43:33.606ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1:0"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::c]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36312024-05-08T03:43:33.606ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36322024-05-08T03:43:33.779ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 42.066444ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36332024-05-08T03:43:33.831ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 145.927872ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36342024-05-08T03:43:34.023ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 153.385655ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36352024-05-08T03:43:34.186ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 442.544438ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36362024-05-08T03:43:34.640ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 530.392116ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36372024-05-08T03:43:35.182ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 1.026767904s file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36382024-05-08T03:43:36.217ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 1.422772866s file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36392024-05-08T03:43:37.650ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 855.592004ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d zone = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36402024-05-08T03:43:37.920ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61319
36412024-05-08T03:43:37.920ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 87 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61319 req_id = 6389a3b4-46f3-42c7-adbe-a0b188d9adeb response_code = 200 uri = /baseboard
36422024-05-08T03:43:38.537ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd file = sled-agent/src/instance.rs:1423 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36432024-05-08T03:43:38.547ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 39.244311ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36442024-05-08T03:43:38.596ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 88.970123ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36452024-05-08T03:43:38.695ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 164.39548ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36462024-05-08T03:43:38.869ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd") failed: Property not found. retry in 377.073053ms file = illumos-utils/src/svc.rs:36 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36472024-05-08T03:43:39.257ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1432 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36482024-05-08T03:43:39.292ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1446 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36492024-05-08T03:43:39.294ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAALcB2oWKhYAABwHahYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAC3AdqFioWAAAcB2oWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJiZGE3ZGVhOS05ZTU0LTQ4MGYtYWNjNS1lNTU3NWY3MTE3MmQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJRExTY1lsdVAvd2RCVCs3L0FJK1M5MW9UbU95WFpTOUVkODJyTDErclhobCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-afd9851865e", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: e8cbe33e-b6b0-4ad4-96b6-1544ea31e55b, read_only_parent: Some(Volume { block_size: 512, id: 2728540c-b0df-45e1-941b-f3cc5a445341, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 738959c6-c771-4d96-955e-1d1120abe8a6, key: Some("ROJ+L4dC8b6WoTWPPXqMnTdQiJo6WFiQzGfCLsVVwrk="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: e8cbe33e-b6b0-4ad4-96b6-1544ea31e55b, key: Some("ENl72cpprsNbdNCXbNY1JquTakKEw0CETtyguYvYHtY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::16]:19002", "[fd00:1122:3344:101::15]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "vopte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: bda7dea9-9e54-480f-acc5-e5575f71172d, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 6120e812-bc0d-472f-9b7c-f86cb8218397, silo_id: 8dac2178-2bd9-4d3d-ae2d-15851d0211c3 }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:685 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36502024-05-08T03:43:39.406ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:687 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36512024-05-08T03:43:39.407ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-05-08T03:43:39.407330894Z } file = sled-agent/src/instance.rs:569 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36522024-05-08T03:43:39.407ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d new_instance_state = InstanceRuntimeState { propolis_id: Some(ef94a682-3038-4280-bef1-5a995ac28fbd), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-08T03:43:31.727598140Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-05-08T03:43:39.407330894Z } propolis_id = ef94a682-3038-4280-bef1-5a995ac28fbd
36532024-05-08T03:43:39.407ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(ef94a682-3038-4280-bef1-5a995ac28fbd), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-08T03:43:31.727598140Z }, propolis_id: ef94a682-3038-4280-bef1-5a995ac28fbd, vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-05-08T03:43:39.407330894Z } }
36542024-05-08T03:43:39.438ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-05-08T03:43:39.438288186Z } file = sled-agent/src/instance.rs:569 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36552024-05-08T03:43:39.438ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d new_instance_state = InstanceRuntimeState { propolis_id: Some(ef94a682-3038-4280-bef1-5a995ac28fbd), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-08T03:43:31.727598140Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-08T03:43:39.438288186Z } propolis_id = ef94a682-3038-4280-bef1-5a995ac28fbd
36562024-05-08T03:43:39.438ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(ef94a682-3038-4280-bef1-5a995ac28fbd), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-08T03:43:31.727598140Z }, propolis_id: ef94a682-3038-4280-bef1-5a995ac28fbd, vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-08T03:43:39.438288186Z } }
36572024-05-08T03:43:49.495ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61234
36582024-05-08T03:43:49.496ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61234 req_id = fe3c0b58-3ba9-493f-b62c-7f8bfd883b6b response_code = 200 uri = /instances/bda7dea9-9e54-480f-acc5-e5575f71172d/state
36592024-05-08T03:43:49.496ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-05-08T03:43:49.496919782Z } file = sled-agent/src/instance.rs:569 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36602024-05-08T03:43:49.496ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d new_instance_state = InstanceRuntimeState { propolis_id: Some(ef94a682-3038-4280-bef1-5a995ac28fbd), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-08T03:43:31.727598140Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-05-08T03:43:49.496919782Z } propolis_id = ef94a682-3038-4280-bef1-5a995ac28fbd
36612024-05-08T03:43:49.496ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(ef94a682-3038-4280-bef1-5a995ac28fbd), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-08T03:43:31.727598140Z }, propolis_id: ef94a682-3038-4280-bef1-5a995ac28fbd, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-05-08T03:43:49.496919782Z } }
36622024-05-08T03:43:49.525ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-05-08T03:43:49.525742541Z } file = sled-agent/src/instance.rs:569 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36632024-05-08T03:43:49.525ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-08T03:43:49.525742541Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-05-08T03:43:49.525742541Z } propolis_id = ef94a682-3038-4280-bef1-5a995ac28fbd
36642024-05-08T03:43:49.525ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:606 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36652024-05-08T03:43:49.525ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
36662024-05-08T03:43:49.526ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
36672024-05-08T03:43:49.526ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/80ce766d-f6eb-440d-b6a8-ad5a1011c02b/debug/bundle/zone", "/pool/int/b3bd066e-6f80-4f80-bc0d-64ec35b22d85/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/58eb44a6-f8c0-46ec-bcea-52bf3efa8996/crypt/debug", "/pool/ext/888a035b-7ecc-4eff-a005-8001c3d3ccff/crypt/debug", "/pool/ext/1afb2a46-5b47-46d7-9926-14b7e21c6407/crypt/debug", "/pool/ext/7c0211ed-cba6-48d8-8b63-a5f21383d980/crypt/debug", "/pool/ext/fc47eb1d-9a62-49dd-845d-82b29d12e969/crypt/debug", "/pool/ext/62a3be27-a8c4-4992-8657-cb4aacfdb8ab/crypt/debug", "/pool/ext/9edc3ed7-0367-4fc4-87ff-888719d0b682/crypt/debug", "/pool/ext/ed53811e-9a91-4270-bd15-bcb96ef98de1/crypt/debug", "/pool/ext/5d9bd5bb-8b9e-4dab-8435-b24d7c2ec621/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:437 zone_name = oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd
36682024-05-08T03:43:50.193ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1230 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd", bundle_id: 0ee97c09-0623-4cf4-96da-f2e0f3789515 }, time_created: 2024-05-08T03:43:49.526846737Z, version: 0, cause: TerminatedInstance }
36692024-05-08T03:43:50.193ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_ef94a682-3038-4280-bef1-5a995ac28fbd file = sled-agent/src/instance.rs:780 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36702024-05-08T03:43:51.953ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:269 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d
36712024-05-08T03:43:51.977ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-08T03:43:49.525742541Z }, propolis_id: ef94a682-3038-4280-bef1-5a995ac28fbd, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-05-08T03:43:51.977842795Z } }
36722024-05-08T03:43:52.074ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33921
36732024-05-08T03:43:52.075ZWARNSledAgent (PortManager): unset_virtual_nic_host unimplmented file = illumos-utils/src/opte/port_manager.rs:620
36742024-05-08T03:43:52.075ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1240 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:33921 req_id = a10455df-db67-40cd-be7d-d67582046cab response_code = 204 uri = /v2p/3cfb1958-3aef-42f5-bea9-40aca6841824
36752024-05-08T03:43:52.356ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-08T03:43:49.525742541Z }, propolis_id: ef94a682-3038-4280-bef1-5a995ac28fbd, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-05-08T03:43:51.977842795Z } }
36762024-05-08T03:43:52.359ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:723 instance_id = bda7dea9-9e54-480f-acc5-e5575f71172d