Max level shown:
1 [ Mar 30 05:42:13 Enabled. ]
2 [ Mar 30 05:42:13 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Mar 30 05:42:13 Method "start" exited with status 0. ]
4 [ Mar 30 05:42:13 Rereading configuration. ]
5 [ Mar 30 05:42:13 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-03-30T05:42:13.723ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:163
82024-03-30T05:42:13.724ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:253
92024-03-30T05:42:13.777ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:260
102024-03-30T05:42:13.777ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-03-30T05:42:13.893ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-03-30T05:42:13.902ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-03-30T05:42:13.910ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:270 path = /var/run/oxide/
142024-03-30T05:42:14.514ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:9fb2::, len: 64 }
152024-03-30T05:42:14.654ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:115
162024-03-30T05:42:14.654ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:127
172024-03-30T05:42:14.655ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:137
182024-03-30T05:42:14.655ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:157 sled_mode = Scrimlet { asic: SoftNpuZone }
192024-03-30T05:42:14.655ZINFOSledAgent (DumpSetup-archival): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:291
202024-03-30T05:42:14.655ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
212024-03-30T05:42:14.655ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:609
222024-03-30T05:42:14.658ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:535
232024-03-30T05:42:14.658ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:658
242024-03-30T05:42:14.658ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:171
252024-03-30T05:42:14.658ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:716
262024-03-30T05:42:14.658ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/m2_0.vdev
272024-03-30T05:42:14.658ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot file = sled-agent/src/hardware_monitor.rs:261 underlay_network_info = None
282024-03-30T05:42:14.658ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }), tx: ... } file = sled-storage/src/manager.rs:381
292024-03-30T05:42:14.658ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
302024-03-30T05:42:14.658ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } path = /scratch/m2_0.vdev
312024-03-30T05:42:14.669ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:301
322024-03-30T05:42:14.669ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:312 id = eaeb58f9-7756-4551-8a4b-dc00787e5906
332024-03-30T05:42:14.768ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal }
342024-03-30T05:42:14.971ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
352024-03-30T05:42:14.971ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/m2_1.vdev
362024-03-30T05:42:14.971ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
372024-03-30T05:42:14.971ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }), tx: ... } file = sled-storage/src/manager.rs:381
382024-03-30T05:42:14.971ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
392024-03-30T05:42:14.971ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } path = /scratch/m2_1.vdev
402024-03-30T05:42:14.979ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:301
412024-03-30T05:42:14.979ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:312 id = f076b946-15f4-4270-9a34-aa6e75797fbb
422024-03-30T05:42:15.066ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal }
432024-03-30T05:42:15.269ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_0.vdev
442024-03-30T05:42:15.269ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
452024-03-30T05:42:15.269ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }), tx: ... } file = sled-storage/src/manager.rs:381
462024-03-30T05:42:15.269ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
472024-03-30T05:42:15.269ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_1.vdev
482024-03-30T05:42:15.269ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
492024-03-30T05:42:15.269ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }), tx: ... } file = sled-storage/src/manager.rs:381
502024-03-30T05:42:15.269ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
512024-03-30T05:42:15.269ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_2.vdev
522024-03-30T05:42:15.269ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
532024-03-30T05:42:15.269ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }), tx: ... } file = sled-storage/src/manager.rs:381
542024-03-30T05:42:15.269ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
552024-03-30T05:42:15.269ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_3.vdev
562024-03-30T05:42:15.269ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }), tx: ... } file = sled-storage/src/manager.rs:381
572024-03-30T05:42:15.269ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
582024-03-30T05:42:15.269ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
592024-03-30T05:42:15.269ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_4.vdev
602024-03-30T05:42:15.269ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }), tx: ... } file = sled-storage/src/manager.rs:381
612024-03-30T05:42:15.269ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
622024-03-30T05:42:15.269ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
632024-03-30T05:42:15.269ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_5.vdev
642024-03-30T05:42:15.269ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }), tx: ... } file = sled-storage/src/manager.rs:381
652024-03-30T05:42:15.269ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
662024-03-30T05:42:15.269ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
672024-03-30T05:42:15.269ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_6.vdev
682024-03-30T05:42:15.269ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }), tx: ... } file = sled-storage/src/manager.rs:381
692024-03-30T05:42:15.269ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
702024-03-30T05:42:15.269ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
712024-03-30T05:42:15.269ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_7.vdev
722024-03-30T05:42:15.269ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }), tx: ... } file = sled-storage/src/manager.rs:381
732024-03-30T05:42:15.270ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
742024-03-30T05:42:15.270ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
752024-03-30T05:42:15.270ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:227 vdev = /scratch/u2_8.vdev
762024-03-30T05:42:15.270ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }), tx: ... } file = sled-storage/src/manager.rs:381
772024-03-30T05:42:15.270ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
782024-03-30T05:42:15.270ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
792024-03-30T05:42:15.270ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:87
802024-03-30T05:42:15.270ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
812024-03-30T05:42:15.270ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" } file = sled-agent/src/long_running_tasks.rs:89
822024-03-30T05:42:15.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
832024-03-30T05:42:15.270ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:195
842024-03-30T05:42:15.270ZINFOSledAgent (bootstore): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
852024-03-30T05:42:15.270ZINFOSledAgent (bootstore): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
862024-03-30T05:42:15.270ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
872024-03-30T05:42:15.270ZINFOSledAgent (bootstore): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
882024-03-30T05:42:15.270ZINFOSledAgent (bootstore): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
892024-03-30T05:42:15.270ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
902024-03-30T05:42:15.270ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:200
912024-03-30T05:42:15.270ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:215
922024-03-30T05:42:15.281ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:701
932024-03-30T05:42:15.281ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 1569, tv_nsec: 716438591 } time_to_next_cleanup = 599.999919109s
942024-03-30T05:42:15.281ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3588
952024-03-30T05:42:15.281ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
962024-03-30T05:42:15.281ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/sled-agent-request.json file = common/src/ledger.rs:188
972024-03-30T05:42:15.283ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/sled-agent-request.json file = common/src/ledger.rs:188
982024-03-30T05:42:15.284ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:195 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
992024-03-30T05:42:15.284ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1002024-03-30T05:42:15.285ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV1 { routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.1.100, }, ], addresses: [ V4( Ipv4Network { addr: 10.151.1.174, prefix: 24, }, ), ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, } file = sled-agent/src/rack_setup/service.rs:997
1012024-03-30T05:42:15.285ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1022024-03-30T05:42:15.285ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1032024-03-30T05:42:15.285ZINFOSledAgent (RSS): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1042024-03-30T05:42:15.285ZINFOSledAgent (RSS): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1052024-03-30T05:42:15.285ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1057
1062024-03-30T05:42:15.285ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1072024-03-30T05:42:15.285ZINFOSledAgent (RSS): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/rss-sled-plan.json file = common/src/ledger.rs:188
1082024-03-30T05:42:15.285ZINFOSledAgent (RSS): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/rss-sled-plan.json file = common/src/ledger.rs:188
1092024-03-30T05:42:15.285ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1098
1102024-03-30T05:42:15.285ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:118
1112024-03-30T05:42:15.285ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:91
1122024-03-30T05:42:15.285ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1132024-03-30T05:42:15.285ZINFOSledAgent (RSS): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1142024-03-30T05:42:15.285ZINFOSledAgent (RSS): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1152024-03-30T05:42:15.285ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:138
1162024-03-30T05:42:15.285ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1131
1172024-03-30T05:42:15.285ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1182024-03-30T05:42:15.285ZINFOSledAgent (bootstore): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1192024-03-30T05:42:15.286ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1202024-03-30T05:42:15.286ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1212024-03-30T05:42:15.286ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 93b0cc91-dc62-481c-9ea1-425d29f058ae, rack_id: 98531190-dbea-48cf-b76b-c50c819faca1, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1222024-03-30T05:42:15.286ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:57598
1232024-03-30T05:42:15.286ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 93b0cc91-dc62-481c-9ea1-425d29f058ae, rack_id: 98531190-dbea-48cf-b76b-c50c819faca1, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344:101::, prefix: 64 }) } } } file = sled-agent/src/bootstrap/server.rs:356
1242024-03-30T05:42:15.286ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:371
1252024-03-30T05:42:15.286ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1262024-03-30T05:42:15.286ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1272024-03-30T05:42:15.286ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:381
1282024-03-30T05:42:15.286ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:530
1292024-03-30T05:42:15.286ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1302024-03-30T05:42:15.286ZINFOSledAgent (StorageManager): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1312024-03-30T05:42:15.286ZINFOSledAgent (StorageManager): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1322024-03-30T05:42:15.286ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:519
1332024-03-30T05:42:15.286ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:547
1342024-03-30T05:42:15.289ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3819
1352024-03-30T05:42:15.298ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
1362024-03-30T05:42:15.308ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
1372024-03-30T05:42:15.317ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
1382024-03-30T05:42:15.326ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
1392024-03-30T05:42:15.334ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
1402024-03-30T05:42:15.343ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
1412024-03-30T05:42:15.352ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
1422024-03-30T05:42:15.360ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
1432024-03-30T05:42:15.369ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
1442024-03-30T05:42:15.369ZINFOSledAgent (StorageManager): No disks to be automatically managed file = sled-storage/src/manager.rs:558
1452024-03-30T05:42:15.369ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1462024-03-30T05:42:15.369ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:370
1472024-03-30T05:42:15.369ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
1482024-03-30T05:42:15.369ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:449 result = DisksManagementResult { status: [] }
1492024-03-30T05:42:15.378ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:355 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
1502024-03-30T05:42:15.378ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1512024-03-30T05:42:15.378ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:367 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
1522024-03-30T05:42:15.378ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1532024-03-30T05:42:15.381ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 64.415612ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c032164d-abdb-41e6-a4b2-78d6ec53ba55", "content-length": "124", "date": "Sat, 30 Mar 2024 05:42:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c032164d-abdb-41e6-a4b2-78d6ec53ba55" } file = clients/ddm-admin-client/src/lib.rs:178
1542024-03-30T05:42:15.389ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:315
1552024-03-30T05:42:15.392ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_eaeb58f9-7756-4551-8a4b-dc00787e5906/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1562024-03-30T05:42:15.407ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1572024-03-30T05:42:15.415ZINFOSledAgent: successfully created encrypted zvol: name="oxi_eaeb58f9-7756-4551-8a4b-dc00787e5906/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1582024-03-30T05:42:15.415ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_eaeb58f9-7756-4551-8a4b-dc00787e5906/swap" file = sled-agent/src/swap_device.rs:96
1592024-03-30T05:42:15.420ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:382 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
1602024-03-30T05:42:15.420ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1612024-03-30T05:42:15.420ZINFOSledAgent (BackingFs): Ensuring dataset oxi_eaeb58f9-7756-4551-8a4b-dc00787e5906/backing/fmd file = sled-agent/src/backing_fs.rs:136
1622024-03-30T05:42:15.446ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1632024-03-30T05:42:15.447ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 152.677356ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7966b747-4356-423c-80c7-7b3ec958e8d4", "content-length": "124", "date": "Sat, 30 Mar 2024 05:42:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7966b747-4356-423c-80c7-7b3ec958e8d4" } file = clients/ddm-admin-client/src/lib.rs:178
1642024-03-30T05:42:15.487ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1652024-03-30T05:42:15.600ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1662024-03-30T05:42:15.601ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 190.739895ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "176f5c2a-e672-4aa7-b646-3bd6c0c1fc69", "content-length": "124", "date": "Sat, 30 Mar 2024 05:42:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "176f5c2a-e672-4aa7-b646-3bd6c0c1fc69" } file = clients/ddm-admin-client/src/lib.rs:178
1672024-03-30T05:42:15.705ZINFOSledAgent (BackingFs): Mounting oxi_eaeb58f9-7756-4551-8a4b-dc00787e5906/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1682024-03-30T05:42:15.718ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1692024-03-30T05:42:15.718ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1702024-03-30T05:42:15.718ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1712024-03-30T05:42:15.718ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1722024-03-30T05:42:15.792ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1732024-03-30T05:42:15.793ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 89.239827ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3319cbd7-7995-457b-b0a7-e04da51e93fa", "content-length": "124", "date": "Sat, 30 Mar 2024 05:42:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3319cbd7-7995-457b-b0a7-e04da51e93fa" } file = clients/ddm-admin-client/src/lib.rs:178
1742024-03-30T05:42:15.883ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1752024-03-30T05:42:15.883ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 230.871811ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e2c41dfb-1832-41d4-bd41-53da445c17e4", "content-length": "124", "date": "Sat, 30 Mar 2024 05:42:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e2c41dfb-1832-41d4-bd41-53da445c17e4" } file = clients/ddm-admin-client/src/lib.rs:178
1762024-03-30T05:42:16.115ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1772024-03-30T05:42:16.116ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 218.346477ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "60db722d-60c7-42c8-9b69-294bb2217509", "content-length": "124", "date": "Sat, 30 Mar 2024 05:42:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "60db722d-60c7-42c8-9b69-294bb2217509" } file = clients/ddm-admin-client/src/lib.rs:178
1782024-03-30T05:42:16.336ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1792024-03-30T05:42:16.336ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 347.876502ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d7f0c78e-3a39-4abb-94dc-b2f2992ff602", "content-length": "124", "date": "Sat, 30 Mar 2024 05:42:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d7f0c78e-3a39-4abb-94dc-b2f2992ff602" } file = clients/ddm-admin-client/src/lib.rs:178
1802024-03-30T05:42:16.430ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
1812024-03-30T05:42:16.476ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:266 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
1822024-03-30T05:42:16.476ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:274 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
1832024-03-30T05:42:16.476ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:983 underlay_address = fd00:1122:3344:101::1
1842024-03-30T05:42:16.476ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1852024-03-30T05:42:16.477ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:195 local_addr = [fd00:1122:3344:101::1]:12345
1862024-03-30T05:42:16.478ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:426
1872024-03-30T05:42:16.478ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1882024-03-30T05:42:16.478ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
1892024-03-30T05:42:16.478ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 332.480585ms sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
1902024-03-30T05:42:16.478ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
1912024-03-30T05:42:16.478ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
1922024-03-30T05:42:16.478ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
1932024-03-30T05:42:16.478ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot file = sled-agent/src/hardware_monitor.rs:261 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV1 { rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV1 { routes: [RouteConfig { destination: V4(Ipv4Network { addr: 0.0.0.0, prefix: 0 }), nexthop: 10.151.1.100 }], addresses: [V4(Ipv4Network { addr: 10.151.1.174, prefix: 24 })], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
1942024-03-30T05:42:16.478ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3588
1952024-03-30T05:42:16.478ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:57598
1962024-03-30T05:42:16.478ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
1972024-03-30T05:42:16.478ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1982024-03-30T05:42:16.678ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1992024-03-30T05:42:16.678ZINFOSledAgent (RSS): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2002024-03-30T05:42:16.678ZINFOSledAgent (RSS): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2012024-03-30T05:42:16.678ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2022024-03-30T05:42:16.678ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2032024-03-30T05:42:16.685ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
2042024-03-30T05:42:16.710ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60256
2052024-03-30T05:42:16.710ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2062024-03-30T05:42:16.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60256 req_id = afafbef2-3118-4fd5-8e0b-9af682093dbe response_code = 200 uri = /inventory
2072024-03-30T05:42:16.741ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55643
2082024-03-30T05:42:16.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 41 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55643 req_id = f62e8fe6-3b21-4b19-9af3-691b6f263288 response_code = 200 uri = /sled-role
2092024-03-30T05:42:16.743ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2102024-03-30T05:42:16.743ZINFOSledAgent (RSS): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2112024-03-30T05:42:16.743ZINFOSledAgent (RSS): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2122024-03-30T05:42:16.743ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:802
2132024-03-30T05:42:16.781ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: bd856e1e-ac7c-42f0-8254-c13876319841, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, pool_id: 82347231-68ab-4934-8007-b28d9a3a3777 }, OmicronPhysicalDiskConfig { id: e284228c-0e5b-4599-8158-4449d9538b60, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, pool_id: 28899eb6-914e-44b6-b77e-9d82d92186d6 }, OmicronPhysicalDiskConfig { id: 447cadf7-31c9-49a1-9141-b94373722b3b, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, pool_id: 6de8542c-123a-4abc-9a88-e72f1063a9f4 }, OmicronPhysicalDiskConfig { id: d0107f86-d382-463c-a450-b342158c84f7, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, pool_id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5 }, OmicronPhysicalDiskConfig { id: 4d836df1-a884-4e12-adcd-7afb4d0e2299, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, pool_id: 5c3019ca-328c-47ab-9982-06e092199db8 }, OmicronPhysicalDiskConfig { id: be27dfe8-0d06-46ec-abb5-1646d8adc826, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, pool_id: 3c92b6bd-8180-45a3-9123-a99ddc649159 }, OmicronPhysicalDiskConfig { id: 47eb38d4-25bb-4df1-94a2-9e15e1e2f8f0, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, pool_id: ea19d3d4-e058-46fb-981b-9ccb255f563c }, OmicronPhysicalDiskConfig { id: a050ef5b-d930-47c7-9e01-49fe59c8434a, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, pool_id: e22df0ad-bbf8-4902-93d6-e67ff23c3234 }, OmicronPhysicalDiskConfig { id: 366f48ff-1929-4f47-a136-197d43fc6605, identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, pool_id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:344 sled_address = [fd00:1122:3344:101::1]:12345
2142024-03-30T05:42:16.781ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58072
2152024-03-30T05:42:16.782ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, id: bd856e1e-ac7c-42f0-8254-c13876319841, pool_id: 82347231-68ab-4934-8007-b28d9a3a3777 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, id: e284228c-0e5b-4599-8158-4449d9538b60, pool_id: 28899eb6-914e-44b6-b77e-9d82d92186d6 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, id: 447cadf7-31c9-49a1-9141-b94373722b3b, pool_id: 6de8542c-123a-4abc-9a88-e72f1063a9f4 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, id: d0107f86-d382-463c-a450-b342158c84f7, pool_id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, id: 4d836df1-a884-4e12-adcd-7afb4d0e2299, pool_id: 5c3019ca-328c-47ab-9982-06e092199db8 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, id: be27dfe8-0d06-46ec-abb5-1646d8adc826, pool_id: 3c92b6bd-8180-45a3-9123-a99ddc649159 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, id: 47eb38d4-25bb-4df1-94a2-9e15e1e2f8f0, pool_id: ea19d3d4-e058-46fb-981b-9ccb255f563c }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, id: a050ef5b-d930-47c7-9e01-49fe59c8434a, pool_id: e22df0ad-bbf8-4902-93d6-e67ff23c3234 }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, id: 366f48ff-1929-4f47-a136-197d43fc6605, pool_id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd }] }, tx: ... } file = sled-storage/src/manager.rs:381
2162024-03-30T05:42:16.782ZINFOSledAgent (StorageManager): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2172024-03-30T05:42:16.782ZINFOSledAgent (StorageManager): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2182024-03-30T05:42:16.782ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:681 request = omicron_physical_disks_ensure
2192024-03-30T05:42:16.782ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
2202024-03-30T05:42:16.782ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2212024-03-30T05:42:16.782ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2222024-03-30T05:42:16.782ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } path = /scratch/u2_0.vdev
2232024-03-30T05:42:16.882ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 712.795655ms sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2242024-03-30T05:42:16.889ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:301
2252024-03-30T05:42:16.889ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 82347231-68ab-4934-8007-b28d9a3a3777
2262024-03-30T05:42:17.579ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2272024-03-30T05:42:17.596ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 1.121519062s sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2282024-03-30T05:42:17.999ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }
2292024-03-30T05:42:18.192ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2302024-03-30T05:42:18.192ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2312024-03-30T05:42:18.192ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2322024-03-30T05:42:18.192ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2332024-03-30T05:42:18.192ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2342024-03-30T05:42:18.192ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2352024-03-30T05:42:18.192ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2362024-03-30T05:42:18.653ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2372024-03-30T05:42:18.653ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2382024-03-30T05:42:18.852ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 2.092164439s sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2392024-03-30T05:42:18.853ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
2402024-03-30T05:42:19.085ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_82347231-68ab-4934-8007-b28d9a3a3777/crypt/zone file = sled-storage/src/dataset.rs:395
2412024-03-30T05:42:19.276ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_switch
2422024-03-30T05:42:20.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 34.760811ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2432024-03-30T05:42:20.126ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2442024-03-30T05:42:20.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 90.579483ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2452024-03-30T05:42:20.769ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2462024-03-30T05:42:21.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 208.44782ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2472024-03-30T05:42:21.445ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2482024-03-30T05:42:21.445ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 4.400028945s sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2492024-03-30T05:42:21.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 579.942719ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2502024-03-30T05:42:22.276ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_82347231-68ab-4934-8007-b28d9a3a3777 file = sled-storage/src/dataset.rs:480
2512024-03-30T05:42:22.697ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2522024-03-30T05:42:22.697ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2532024-03-30T05:42:22.697ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2542024-03-30T05:42:22.697ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2552024-03-30T05:42:22.697ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } path = /scratch/u2_1.vdev
2562024-03-30T05:42:22.905ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2572024-03-30T05:42:22.905ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2582024-03-30T05:42:22.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.118681489s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2592024-03-30T05:42:22.911ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:301
2602024-03-30T05:42:22.911ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 28899eb6-914e-44b6-b77e-9d82d92186d6
2612024-03-30T05:42:23.978ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2622024-03-30T05:42:23.989ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }
2632024-03-30T05:42:24.427ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2642024-03-30T05:42:24.427ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2652024-03-30T05:42:24.427ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2662024-03-30T05:42:24.427ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2672024-03-30T05:42:24.427ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2682024-03-30T05:42:24.429ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2692024-03-30T05:42:24.429ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2702024-03-30T05:42:24.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.358620191s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2712024-03-30T05:42:24.906ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2722024-03-30T05:42:25.131ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2732024-03-30T05:42:25.133ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
2742024-03-30T05:42:25.377ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_28899eb6-914e-44b6-b77e-9d82d92186d6/crypt/zone file = sled-storage/src/dataset.rs:395
2752024-03-30T05:42:26.501ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2762024-03-30T05:42:26.501ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/metrics/producers): error trying to connect: dns error: proto error: io error: No route to host (os error 148)" file = sled-agent/src/sled_agent.rs:1178 retry_after = 10.589991045s sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2772024-03-30T05:42:26.941ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:278 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2782024-03-30T05:42:26.943ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
2792024-03-30T05:42:27.024ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_28899eb6-914e-44b6-b77e-9d82d92186d6 file = sled-storage/src/dataset.rs:480
2802024-03-30T05:42:27.033ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2812024-03-30T05:42:27.033ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
2822024-03-30T05:42:27.033ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
2832024-03-30T05:42:27.033ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2842024-03-30T05:42:27.033ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } path = /scratch/u2_2.vdev
2852024-03-30T05:42:27.042ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:301
2862024-03-30T05:42:27.043ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 6de8542c-123a-4abc-9a88-e72f1063a9f4
2872024-03-30T05:42:27.136ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }
2882024-03-30T05:42:27.143ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
2892024-03-30T05:42:27.143ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
2902024-03-30T05:42:27.143ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
2912024-03-30T05:42:27.144ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
2922024-03-30T05:42:27.144ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
2932024-03-30T05:42:27.144ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
2942024-03-30T05:42:27.144ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2952024-03-30T05:42:27.178ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
2962024-03-30T05:42:27.186ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_6de8542c-123a-4abc-9a88-e72f1063a9f4/crypt/zone file = sled-storage/src/dataset.rs:395
2972024-03-30T05:42:27.309ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_6de8542c-123a-4abc-9a88-e72f1063a9f4 file = sled-storage/src/dataset.rs:480
2982024-03-30T05:42:27.317ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
2992024-03-30T05:42:27.317ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3002024-03-30T05:42:27.317ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3012024-03-30T05:42:27.317ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3022024-03-30T05:42:27.317ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } path = /scratch/u2_3.vdev
3032024-03-30T05:42:27.327ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:301
3042024-03-30T05:42:27.327ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = e43353e3-7e66-4e6f-bd56-19f8143ee8b5
3052024-03-30T05:42:27.425ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }
3062024-03-30T05:42:27.433ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3072024-03-30T05:42:27.434ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3082024-03-30T05:42:27.434ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3092024-03-30T05:42:27.434ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3102024-03-30T05:42:27.434ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3112024-03-30T05:42:27.434ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3122024-03-30T05:42:27.434ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3132024-03-30T05:42:27.470ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3142024-03-30T05:42:27.478ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e43353e3-7e66-4e6f-bd56-19f8143ee8b5/crypt/zone file = sled-storage/src/dataset.rs:395
3152024-03-30T05:42:27.505ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
3162024-03-30T05:42:27.602ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e43353e3-7e66-4e6f-bd56-19f8143ee8b5 file = sled-storage/src/dataset.rs:480
3172024-03-30T05:42:27.610ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3182024-03-30T05:42:27.610ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3192024-03-30T05:42:27.610ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3202024-03-30T05:42:27.610ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3212024-03-30T05:42:27.610ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } path = /scratch/u2_4.vdev
3222024-03-30T05:42:27.620ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:301
3232024-03-30T05:42:27.620ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 5c3019ca-328c-47ab-9982-06e092199db8
3242024-03-30T05:42:27.714ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }
3252024-03-30T05:42:27.722ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3262024-03-30T05:42:27.722ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3272024-03-30T05:42:27.722ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3282024-03-30T05:42:27.722ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3292024-03-30T05:42:27.722ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3302024-03-30T05:42:27.722ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3312024-03-30T05:42:27.722ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3322024-03-30T05:42:27.757ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3332024-03-30T05:42:27.765ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_5c3019ca-328c-47ab-9982-06e092199db8/crypt/zone file = sled-storage/src/dataset.rs:395
3342024-03-30T05:42:27.889ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_5c3019ca-328c-47ab-9982-06e092199db8 file = sled-storage/src/dataset.rs:480
3352024-03-30T05:42:27.897ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3362024-03-30T05:42:27.897ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3372024-03-30T05:42:27.897ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3382024-03-30T05:42:27.897ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3392024-03-30T05:42:27.897ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } path = /scratch/u2_5.vdev
3402024-03-30T05:42:27.906ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:301
3412024-03-30T05:42:27.906ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = 3c92b6bd-8180-45a3-9123-a99ddc649159
3422024-03-30T05:42:28.000ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }
3432024-03-30T05:42:28.008ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3442024-03-30T05:42:28.008ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3452024-03-30T05:42:28.008ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3462024-03-30T05:42:28.008ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3472024-03-30T05:42:28.008ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3482024-03-30T05:42:28.009ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3492024-03-30T05:42:28.009ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3502024-03-30T05:42:28.046ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3512024-03-30T05:42:28.054ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_3c92b6bd-8180-45a3-9123-a99ddc649159/crypt/zone file = sled-storage/src/dataset.rs:395
3522024-03-30T05:42:28.175ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_3c92b6bd-8180-45a3-9123-a99ddc649159 file = sled-storage/src/dataset.rs:480
3532024-03-30T05:42:28.183ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3542024-03-30T05:42:28.183ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3552024-03-30T05:42:28.183ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3562024-03-30T05:42:28.183ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3572024-03-30T05:42:28.183ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } path = /scratch/u2_6.vdev
3582024-03-30T05:42:28.192ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:301
3592024-03-30T05:42:28.192ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = ea19d3d4-e058-46fb-981b-9ccb255f563c
3602024-03-30T05:42:28.287ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }
3612024-03-30T05:42:28.295ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3622024-03-30T05:42:28.295ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3632024-03-30T05:42:28.295ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3642024-03-30T05:42:28.295ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3652024-03-30T05:42:28.295ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3662024-03-30T05:42:28.295ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3672024-03-30T05:42:28.295ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3682024-03-30T05:42:28.331ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3692024-03-30T05:42:28.339ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ea19d3d4-e058-46fb-981b-9ccb255f563c/crypt/zone file = sled-storage/src/dataset.rs:395
3702024-03-30T05:42:28.459ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ea19d3d4-e058-46fb-981b-9ccb255f563c file = sled-storage/src/dataset.rs:480
3712024-03-30T05:42:28.467ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3722024-03-30T05:42:28.467ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3732024-03-30T05:42:28.467ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3742024-03-30T05:42:28.467ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3752024-03-30T05:42:28.467ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } path = /scratch/u2_7.vdev
3762024-03-30T05:42:28.476ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:301
3772024-03-30T05:42:28.476ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = e22df0ad-bbf8-4902-93d6-e67ff23c3234
3782024-03-30T05:42:28.570ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }
3792024-03-30T05:42:28.577ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3802024-03-30T05:42:28.577ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3812024-03-30T05:42:28.577ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3822024-03-30T05:42:28.577ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3832024-03-30T05:42:28.577ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3842024-03-30T05:42:28.578ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
3852024-03-30T05:42:28.578ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3862024-03-30T05:42:28.582ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
3872024-03-30T05:42:28.614ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
3882024-03-30T05:42:28.621ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e22df0ad-bbf8-4902-93d6-e67ff23c3234/crypt/zone file = sled-storage/src/dataset.rs:395
3892024-03-30T05:42:28.743ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e22df0ad-bbf8-4902-93d6-e67ff23c3234 file = sled-storage/src/dataset.rs:480
3902024-03-30T05:42:28.751ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
3912024-03-30T05:42:28.751ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
3922024-03-30T05:42:28.751ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:379
3932024-03-30T05:42:28.751ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3942024-03-30T05:42:28.751ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:111 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } path = /scratch/u2_8.vdev
3952024-03-30T05:42:28.761ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:301
3962024-03-30T05:42:28.761ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:307 id = c7e0e489-6c2b-472e-8a1f-efe83bb548dd
3972024-03-30T05:42:28.858ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }
3982024-03-30T05:42:28.866ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/dataset.rs:332
3992024-03-30T05:42:28.866ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:334
4002024-03-30T05:42:28.866ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:338
4012024-03-30T05:42:28.866ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } epoch = 0 file = sled-storage/src/dataset.rs:340
4022024-03-30T05:42:28.866ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4032024-03-30T05:42:28.866ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:38 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4042024-03-30T05:42:28.866ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4052024-03-30T05:42:28.903ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:57
4062024-03-30T05:42:28.911ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c7e0e489-6c2b-472e-8a1f-efe83bb548dd/crypt/zone file = sled-storage/src/dataset.rs:395
4072024-03-30T05:42:28.945ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
4082024-03-30T05:42:28.967ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2286
4092024-03-30T05:42:29.034ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c7e0e489-6c2b-472e-8a1f-efe83bb548dd file = sled-storage/src/dataset.rs:480
4102024-03-30T05:42:29.041ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:477
4112024-03-30T05:42:29.041ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" } file = sled-storage/src/resources.rs:394
4122024-03-30T05:42:29.041ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4132024-03-30T05:42:29.041ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: eaeb58f9-7756-4551-8a4b-dc00787e5906, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }, variant: U2, slot: 1026 }, zpool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: f076b946-15f4-4270-9a34-aa6e75797fbb, kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }, variant: U2, slot: 1027 }, zpool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }, variant: U2, slot: 1028 }, zpool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }, variant: U2, slot: 1029 }, zpool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }, variant: U2, slot: 1030 }, zpool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }, variant: U2, slot: 1031 }, zpool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }, variant: U2, slot: 1032 }, zpool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }, variant: U2, slot: 1033 }, zpool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } })), DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }, variant: U2, slot: 1034 }, zpool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
4142024-03-30T05:42:29.041ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4152024-03-30T05:42:29.041ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 12260047 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:58072 req_id = 31715b1b-3c72-42f2-87a9-5aaa0cf97fd5 response_code = 200 uri = /omicron-physical-disks
4162024-03-30T05:42:29.072ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
4172024-03-30T05:42:29.072ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:46225
4182024-03-30T05:42:29.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4192024-03-30T05:42:29.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
4202024-03-30T05:42:29.123ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4212024-03-30T05:42:29.123ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
4222024-03-30T05:42:29.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4232024-03-30T05:42:29.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
4242024-03-30T05:42:29.222ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4252024-03-30T05:42:29.222ZINFOSledAgent (ServiceManager): No ledger in /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:188
4262024-03-30T05:42:29.222ZINFOSledAgent (ServiceManager): No ledger in /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:188
4272024-03-30T05:42:29.222ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4282024-03-30T05:42:29.267ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4292024-03-30T05:42:29.267ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4302024-03-30T05:42:29.267ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4312024-03-30T05:42:29.315ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795 file = illumos-utils/src/zone.rs:315
4322024-03-30T05:42:29.319ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490 file = illumos-utils/src/zone.rs:315
4332024-03-30T05:42:29.323ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2 file = illumos-utils/src/zone.rs:315
4342024-03-30T05:42:29.334ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795 file = illumos-utils/src/zone.rs:354
4352024-03-30T05:42:29.339ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490 file = illumos-utils/src/zone.rs:354
4362024-03-30T05:42:29.343ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2 file = illumos-utils/src/zone.rs:354
4372024-03-30T05:42:29.585ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
4382024-03-30T05:42:30.586ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
4392024-03-30T05:42:30.944ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
4402024-03-30T05:42:31.700ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4412024-03-30T05:42:31.700ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
4422024-03-30T05:42:31.701ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4432024-03-30T05:42:32.185ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4442024-03-30T05:42:32.185ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
4452024-03-30T05:42:32.667ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4462024-03-30T05:42:32.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 40.811764ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4472024-03-30T05:42:32.675ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4482024-03-30T05:42:32.678ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
4492024-03-30T05:42:32.679ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4502024-03-30T05:42:32.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 63.858422ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4512024-03-30T05:42:32.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 140.797668ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4522024-03-30T05:42:32.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 58.25315ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4532024-03-30T05:42:32.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 61.420802ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4542024-03-30T05:42:32.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 131.022896ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4552024-03-30T05:42:32.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 273.851454ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4562024-03-30T05:42:33.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 63.50206ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4572024-03-30T05:42:33.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 267.54727ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4582024-03-30T05:42:33.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 212.993567ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4592024-03-30T05:42:33.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 518.237125ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4602024-03-30T05:42:33.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 455.712682ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4612024-03-30T05:42:33.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 383.307162ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4622024-03-30T05:42:33.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 507.540896ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4632024-03-30T05:42:33.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 1.238923272s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4642024-03-30T05:42:33.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 468.603195ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4652024-03-30T05:42:34.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 745.802632ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4662024-03-30T05:42:34.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 858.317213ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4672024-03-30T05:42:35.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 774.953847ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4682024-03-30T05:42:35.049ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:9fb2::2 exists in switch zone file = sled-agent/src/services.rs:2303
4692024-03-30T05:42:35.049ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:626 zone = oxz_switch
4702024-03-30T05:42:35.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 706.864869ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4712024-03-30T05:42:35.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 713.253376ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4722024-03-30T05:42:35.608ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fe01:db42 file = sled-agent/src/services.rs:2310
4732024-03-30T05:42:35.615ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2371
4742024-03-30T05:42:35.740ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2556
4752024-03-30T05:42:35.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 830.303914ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4762024-03-30T05:42:35.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 611.299086ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4772024-03-30T05:42:35.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 1.025449849s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4782024-03-30T05:42:36.000ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2720
4792024-03-30T05:42:36.200ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2435
4802024-03-30T05:42:36.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 603.302096ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4812024-03-30T05:42:36.523ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2470
4822024-03-30T05:42:36.615ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 1.319771206s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4832024-03-30T05:42:36.743ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2758
4842024-03-30T05:42:36.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 1.271727719s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
4852024-03-30T05:42:36.931ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2785
4862024-03-30T05:42:37.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 848.769965ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4872024-03-30T05:42:37.142ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2664
4882024-03-30T05:42:37.311ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2464
4892024-03-30T05:42:37.321ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3843 new = [fd00:1122:3344:101::2, ::1] old = [::1]
4902024-03-30T05:42:37.321ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3859
4912024-03-30T05:42:37.321ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:609 zone = oxz_switch
4922024-03-30T05:42:37.867ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3867
4932024-03-30T05:42:37.873ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3923
4942024-03-30T05:42:37.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 1.088799286s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
4952024-03-30T05:42:37.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 1.446633536s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
4962024-03-30T05:42:38.005ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:3985
4972024-03-30T05:42:38.101ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3966 rack_subnet = fd00:1122:3344::
4982024-03-30T05:42:38.122ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4008
4992024-03-30T05:42:38.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 1.301122391s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
5002024-03-30T05:42:38.217ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4035
5012024-03-30T05:42:38.342ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:364
5022024-03-30T05:42:38.374ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:412
5032024-03-30T05:42:38.406ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:589
5042024-03-30T05:42:38.406ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:627
5052024-03-30T05:42:38.406ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:630 uptime = 3
5062024-03-30T05:42:38.406ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:434
5072024-03-30T05:42:38.483ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
5082024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-M2" }
5092024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-0", model: "synthetic-model-U2" }
5102024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-M2" }
5112024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-1", model: "synthetic-model-U2" }
5122024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-2", model: "synthetic-model-U2" }
5132024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-3", model: "synthetic-model-U2" }
5142024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-4", model: "synthetic-model-U2" }
5152024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-5", model: "synthetic-model-U2" }
5162024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-6", model: "synthetic-model-U2" }
5172024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-7", model: "synthetic-model-U2" }
5182024-03-30T05:42:38.483ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", serial: "synthetic-serial-8", model: "synthetic-model-U2" }
5192024-03-30T05:42:39.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795") failed: Property not found. retry in 740.087051ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
5202024-03-30T05:42:39.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490") failed: Property not found. retry in 880.315473ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
5212024-03-30T05:42:39.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2") failed: Property not found. retry in 1.073940868s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
5222024-03-30T05:42:40.610ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_internal_dns_55e16fed-0b39-4c9e-94aa-dbf1dfed9795
5232024-03-30T05:42:40.610ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_internal_dns_c037c9e2-43a2-432a-bcd7-352e08f26490
5242024-03-30T05:42:40.610ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_internal_dns_655db966-87f1-48aa-8868-000f036af1b2
5252024-03-30T05:42:40.610ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5262024-03-30T05:42:40.610ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5272024-03-30T05:42:40.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 11537666 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:46225 req_id = 8a190884-8f3c-4426-9051-8e939640cc87 response_code = 204 uri = /omicron-zones
5282024-03-30T05:42:40.612ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:545
5292024-03-30T05:42:40.856ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 52.200636ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:565
5302024-03-30T05:42:40.910ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 83.757735ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:565
5312024-03-30T05:42:41.046ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:545
5322024-03-30T05:42:41.124ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:545
5332024-03-30T05:42:41.206ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:581
5342024-03-30T05:42:41.206ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5352024-03-30T05:42:41.237ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5362024-03-30T05:42:41.237ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5372024-03-30T05:42:41.237ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5382024-03-30T05:42:41.266ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
5392024-03-30T05:42:41.266ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53993
5402024-03-30T05:42:41.266ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5412024-03-30T05:42:41.266ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
5422024-03-30T05:42:41.286ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5432024-03-30T05:42:41.286ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
5442024-03-30T05:42:41.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5452024-03-30T05:42:41.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
5462024-03-30T05:42:41.324ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5472024-03-30T05:42:41.324ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
5482024-03-30T05:42:41.324ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
5492024-03-30T05:42:41.324ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5502024-03-30T05:42:41.336ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
5512024-03-30T05:42:41.367ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
5522024-03-30T05:42:41.368ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
5532024-03-30T05:42:41.368ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
5542024-03-30T05:42:41.411ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 143, 167]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
5552024-03-30T05:42:41.411ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1277 zone_type = ntp
5562024-03-30T05:42:41.411ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
5572024-03-30T05:42:41.411ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
5582024-03-30T05:42:41.412ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
5592024-03-30T05:42:41.414ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5602024-03-30T05:42:41.439ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 file = illumos-utils/src/zone.rs:315
5612024-03-30T05:42:41.459ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 file = illumos-utils/src/zone.rs:354
5622024-03-30T05:42:41.819ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36446
5632024-03-30T05:42:41.820ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 77 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36446 req_id = aa7f435c-6b86-4065-9126-90715890719d response_code = 200 uri = /baseboard
5642024-03-30T05:42:42.631ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5652024-03-30T05:42:42.632ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5662024-03-30T05:42:42.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 31.653846ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5672024-03-30T05:42:42.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 144.713482ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5682024-03-30T05:42:42.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 251.324635ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5692024-03-30T05:42:43.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 457.55305ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5702024-03-30T05:42:43.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 979.839814ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5712024-03-30T05:42:44.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 714.503579ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5722024-03-30T05:42:45.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 903.89542ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5732024-03-30T05:42:46.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 1.06299245s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5742024-03-30T05:42:47.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 516.508395ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5752024-03-30T05:42:47.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7") failed: Property not found. retry in 1.343568085s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5762024-03-30T05:42:49.286ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7
5772024-03-30T05:42:49.286ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5782024-03-30T05:42:49.286ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5792024-03-30T05:42:49.286ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8020471 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53993 req_id = 4cc65d6a-5c73-4dc3-ae4a-5ba7cea49cf5 response_code = 204 uri = /omicron-zones
5802024-03-30T05:42:49.288ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:622
5812024-03-30T05:42:49.317ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5822024-03-30T05:42:49.317ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51691
5832024-03-30T05:42:49.325ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3574
5842024-03-30T05:42:49.326ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 8587 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51691 req_id = aaeaf4a8-337e-41c0-9f5b-35cfc3c37d41 response_code = 500 uri = /timesync
5852024-03-30T05:42:49.327ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5862024-03-30T05:42:49.486ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5872024-03-30T05:42:49.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65502
5882024-03-30T05:42:53.494ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3574
5892024-03-30T05:42:53.494ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 4007593 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65502 req_id = 4244cfa3-803d-4bfd-b5e3-cd5032f8aa55 response_code = 500 uri = /timesync
5902024-03-30T05:42:53.497ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5912024-03-30T05:42:53.649ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5922024-03-30T05:42:53.649ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47720
5932024-03-30T05:42:53.658ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3574
5942024-03-30T05:42:53.659ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 9082 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47720 req_id = 819e485c-1545-4866-a6b0-9ea264430274 response_code = 500 uri = /timesync
5952024-03-30T05:42:53.662ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
5962024-03-30T05:42:53.848ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
5972024-03-30T05:42:53.848ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60365
5982024-03-30T05:42:57.857ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_2168e6f2-fbb0-4d6d-8811-9fd75847a2a7': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3574
5992024-03-30T05:42:57.857ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 4008451 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60365 req_id = 58428d31-12a9-4d0e-b7ab-8d5b77336fef response_code = 500 uri = /timesync
6002024-03-30T05:42:57.860ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6012024-03-30T05:42:57.997ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6022024-03-30T05:42:57.997ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39270
6032024-03-30T05:42:58.005ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39270 req_id = 9838962e-fe87-442a-adfd-42f257f334b9 response_code = 200 uri = /timesync
6042024-03-30T05:42:58.007ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711777373.8469317, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6052024-03-30T05:42:58.007ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6062024-03-30T05:42:58.246ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6072024-03-30T05:42:58.246ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53043
6082024-03-30T05:42:58.255ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8320 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:53043 req_id = 97f80927-f5ee-4e21-a2c5-6c2eb2701f57 response_code = 200 uri = /timesync
6092024-03-30T05:42:58.258ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711777373.8469317, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6102024-03-30T05:42:58.258ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6112024-03-30T05:42:58.542ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6122024-03-30T05:42:58.543ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54815
6132024-03-30T05:42:58.551ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8720 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54815 req_id = cbe6f74d-e482-40c9-8e45-cf60ce71b022 response_code = 200 uri = /timesync
6142024-03-30T05:42:58.553ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711777373.8469317, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6152024-03-30T05:42:58.553ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6162024-03-30T05:42:58.810ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6172024-03-30T05:42:58.810ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37167
6182024-03-30T05:42:58.818ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 7962 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37167 req_id = 4b4dfd81-5131-4ef1-832e-46879cd169b2 response_code = 200 uri = /timesync
6192024-03-30T05:42:58.820ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711777373.8469317, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6202024-03-30T05:42:58.820ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6212024-03-30T05:42:59.289ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6222024-03-30T05:42:59.289ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38379
6232024-03-30T05:42:59.298ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8051 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38379 req_id = 051f234a-16a3-4518-800b-c58ac496b72c response_code = 200 uri = /timesync
6242024-03-30T05:42:59.300ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711777373.8469317, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6252024-03-30T05:42:59.300ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6262024-03-30T05:42:59.673ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6272024-03-30T05:42:59.673ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51457
6282024-03-30T05:42:59.681ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 7989 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51457 req_id = be436a97-ca58-4b2e-b542-5ba71535f39e response_code = 200 uri = /timesync
6292024-03-30T05:42:59.684ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711777373.8469317, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6302024-03-30T05:42:59.684ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6312024-03-30T05:43:00.042ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6322024-03-30T05:43:00.042ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65458
6332024-03-30T05:43:00.050ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 8057 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65458 req_id = ca934cb7-099b-4dc0-9b53-b8aa8fb29ad4 response_code = 200 uri = /timesync
6342024-03-30T05:43:00.054ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1711777373.8469317, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6352024-03-30T05:43:00.054ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:652
6362024-03-30T05:43:01.476ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:602
6372024-03-30T05:43:01.476ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52940
6382024-03-30T05:43:01.479ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
6392024-03-30T05:43:01.495ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 19503 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52940 req_id = 9a1c80ea-9022-455c-9944-f0288f7180e8 response_code = 200 uri = /timesync
6402024-03-30T05:43:01.497ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 843850428, ip_addr: 50.76.34.188, stratum: 2, ref_time: 1711777380.5411906, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:630
6412024-03-30T05:43:01.497ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1210
6422024-03-30T05:43:01.526ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
6432024-03-30T05:43:01.526ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48593
6442024-03-30T05:43:01.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6452024-03-30T05:43:01.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
6462024-03-30T05:43:01.545ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6472024-03-30T05:43:01.545ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
6482024-03-30T05:43:01.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6492024-03-30T05:43:01.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
6502024-03-30T05:43:01.583ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6512024-03-30T05:43:01.583ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
6522024-03-30T05:43:01.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6532024-03-30T05:43:01.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
6542024-03-30T05:43:01.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6552024-03-30T05:43:01.734ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
6562024-03-30T05:43:01.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6572024-03-30T05:43:01.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
6582024-03-30T05:43:01.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6592024-03-30T05:43:01.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
6602024-03-30T05:43:01.958ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6612024-03-30T05:43:01.958ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
6622024-03-30T05:43:01.958ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
6632024-03-30T05:43:01.958ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6642024-03-30T05:43:01.991ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
6652024-03-30T05:43:01.991ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
6662024-03-30T05:43:01.991ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
6672024-03-30T05:43:01.991ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
6682024-03-30T05:43:01.991ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
6692024-03-30T05:43:01.991ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465", TimeNotSynchronized), ("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349", TimeNotSynchronized), ("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198", TimeNotSynchronized), ("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306", TimeNotSynchronized), ("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 465280 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48593 req_id = eeaa1cae-8957-4cf0-976c-65edc5fdfbe7 response_code = 500 uri = /omicron-zones
6702024-03-30T05:43:01.991ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "eeaa1cae-8957-4cf0-976c-65edc5fdfbe7", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:01 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "eeaa1cae-8957-4cf0-976c-65edc5fdfbe7" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 98.048812ms sled_address = [fd00:1122:3344:101::1]:12345
6712024-03-30T05:43:02.091ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
6722024-03-30T05:43:02.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6732024-03-30T05:43:02.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
6742024-03-30T05:43:02.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6752024-03-30T05:43:02.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
6762024-03-30T05:43:02.128ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6772024-03-30T05:43:02.128ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
6782024-03-30T05:43:02.145ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6792024-03-30T05:43:02.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
6802024-03-30T05:43:02.165ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6812024-03-30T05:43:02.165ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
6822024-03-30T05:43:02.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6832024-03-30T05:43:02.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
6842024-03-30T05:43:02.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6852024-03-30T05:43:02.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
6862024-03-30T05:43:02.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6872024-03-30T05:43:02.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
6882024-03-30T05:43:02.237ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6892024-03-30T05:43:02.237ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
6902024-03-30T05:43:02.237ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
6912024-03-30T05:43:02.237ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6922024-03-30T05:43:02.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
6932024-03-30T05:43:02.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
6942024-03-30T05:43:02.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
6952024-03-30T05:43:02.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
6962024-03-30T05:43:02.270ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
6972024-03-30T05:43:02.270ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198", TimeNotSynchronized), ("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306", TimeNotSynchronized), ("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465", TimeNotSynchronized), ("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f", TimeNotSynchronized), ("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 179033 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48593 req_id = 783397ea-524e-4639-ad29-34566a34b49f response_code = 500 uri = /omicron-zones
6982024-03-30T05:43:02.270ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "783397ea-524e-4639-ad29-34566a34b49f", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:01 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "783397ea-524e-4639-ad29-34566a34b49f" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 137.241429ms sled_address = [fd00:1122:3344:101::1]:12345
6992024-03-30T05:43:02.408ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
7002024-03-30T05:43:02.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7012024-03-30T05:43:02.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
7022024-03-30T05:43:02.427ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7032024-03-30T05:43:02.427ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
7042024-03-30T05:43:02.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
7052024-03-30T05:43:02.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
7062024-03-30T05:43:02.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7072024-03-30T05:43:02.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
7082024-03-30T05:43:02.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7092024-03-30T05:43:02.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
7102024-03-30T05:43:02.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7112024-03-30T05:43:02.498ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
7122024-03-30T05:43:02.515ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7132024-03-30T05:43:02.515ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
7142024-03-30T05:43:02.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
7152024-03-30T05:43:02.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
7162024-03-30T05:43:02.007ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7172024-03-30T05:43:02.007ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
7182024-03-30T05:43:02.007ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
7192024-03-30T05:43:02.007ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7202024-03-30T05:43:02.010ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
7212024-03-30T05:43:02.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7222024-03-30T05:43:02.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7232024-03-30T05:43:02.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7242024-03-30T05:43:02.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7252024-03-30T05:43:02.089ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7262024-03-30T05:43:02.178ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f file = illumos-utils/src/zone.rs:315
7272024-03-30T05:43:02.182ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465 file = illumos-utils/src/zone.rs:315
7282024-03-30T05:43:02.187ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349 file = illumos-utils/src/zone.rs:315
7292024-03-30T05:43:02.191ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198 file = illumos-utils/src/zone.rs:315
7302024-03-30T05:43:02.196ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306 file = illumos-utils/src/zone.rs:315
7312024-03-30T05:43:02.200ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f file = illumos-utils/src/zone.rs:354
7322024-03-30T05:43:02.204ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465 file = illumos-utils/src/zone.rs:354
7332024-03-30T05:43:02.209ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349 file = illumos-utils/src/zone.rs:354
7342024-03-30T05:43:02.213ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198 file = illumos-utils/src/zone.rs:354
7352024-03-30T05:43:02.218ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306 file = illumos-utils/src/zone.rs:354
7362024-03-30T05:43:05.963ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::4"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::4"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7372024-03-30T05:43:05.963ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7382024-03-30T05:43:05.966ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
7392024-03-30T05:43:05.969ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::3"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::3"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7402024-03-30T05:43:05.969ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7412024-03-30T05:43:06.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 74.98675ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7422024-03-30T05:43:06.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 45.151757ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7432024-03-30T05:43:06.161ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::6"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7442024-03-30T05:43:06.161ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7452024-03-30T05:43:06.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 83.345683ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7462024-03-30T05:43:06.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 87.058181ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7472024-03-30T05:43:06.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 201.314312ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7482024-03-30T05:43:06.364ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::5"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7492024-03-30T05:43:06.365ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7502024-03-30T05:43:06.386ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::7"/> <propval type="astring" name="listen_port" value="32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7512024-03-30T05:43:06.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 189.714903ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7522024-03-30T05:43:06.519ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7532024-03-30T05:43:06.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 470.906115ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7542024-03-30T05:43:06.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 40.57049ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7552024-03-30T05:43:06.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 261.241756ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7562024-03-30T05:43:06.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 121.406801ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7572024-03-30T05:43:07.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 798.772729ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7582024-03-30T05:43:07.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 113.975802ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7592024-03-30T05:43:07.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 41.497543ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7602024-03-30T05:43:07.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 690.772957ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7612024-03-30T05:43:07.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 60.376009ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7622024-03-30T05:43:07.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 50.889724ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7632024-03-30T05:43:07.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 418.014792ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7642024-03-30T05:43:07.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 133.974657ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7652024-03-30T05:43:07.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 83.771439ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7662024-03-30T05:43:07.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 514.750444ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7672024-03-30T05:43:07.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.295862068s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7682024-03-30T05:43:07.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 131.981885ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7692024-03-30T05:43:07.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 662.602625ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7702024-03-30T05:43:08.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 1.029241152s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7712024-03-30T05:43:08.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 242.307082ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7722024-03-30T05:43:08.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 691.111938ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7732024-03-30T05:43:08.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.014682526s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7742024-03-30T05:43:08.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 1.156044005s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7752024-03-30T05:43:09.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 1.252827806s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7762024-03-30T05:43:09.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.374148265s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7772024-03-30T05:43:09.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 843.35988ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7782024-03-30T05:43:09.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 657.966386ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7792024-03-30T05:43:09.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 1.098610591s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7802024-03-30T05:43:10.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 685.393483ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7812024-03-30T05:43:10.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 729.104776ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7822024-03-30T05:43:10.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 540.587494ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7832024-03-30T05:43:10.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 915.432079ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7842024-03-30T05:43:10.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 836.161216ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7852024-03-30T05:43:10.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 1.185840432s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7862024-03-30T05:43:10.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 1.475314142s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7872024-03-30T05:43:10.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.171772371s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7882024-03-30T05:43:11.488ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 648.46544ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7892024-03-30T05:43:11.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 837.432914ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7902024-03-30T05:43:12.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 622.382507ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7912024-03-30T05:43:12.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.159309292s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
7922024-03-30T05:43:12.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.017883099s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7932024-03-30T05:43:12.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 961.919317ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
7942024-03-30T05:43:12.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 826.202601ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
7952024-03-30T05:43:12.543ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64015
7962024-03-30T05:43:12.543ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 76 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64015 req_id = e55e0489-0783-41c6-82af-6991f8d16b8c response_code = 200 uri = /baseboard
7972024-03-30T05:43:12.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 662.084947ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
7982024-03-30T05:43:13.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.022508166s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
7992024-03-30T05:43:13.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 651.502242ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8002024-03-30T05:43:13.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.498419787s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8012024-03-30T05:43:13.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 566.447217ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
8022024-03-30T05:43:13.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 1.195305009s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
8032024-03-30T05:43:13.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 509.492856ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8042024-03-30T05:43:13.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198") failed: Property not found. retry in 1.167490939s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
8052024-03-30T05:43:14.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 507.968133ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8062024-03-30T05:43:14.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 1.242041898s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8072024-03-30T05:43:14.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349") failed: Property not found. retry in 742.452379ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
8082024-03-30T05:43:14.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 505.294437ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8092024-03-30T05:43:14.829ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 986.356327ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8102024-03-30T05:43:15.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 535.061675ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8112024-03-30T05:43:15.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 654.650963ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8122024-03-30T05:43:15.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 865.161808ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8132024-03-30T05:43:15.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.149466414s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8142024-03-30T05:43:16.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 826.346364ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8152024-03-30T05:43:16.673ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 943.234757ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8162024-03-30T05:43:17.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.128632566s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8172024-03-30T05:43:17.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 1.228582519s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8182024-03-30T05:43:17.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.184607809s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8192024-03-30T05:43:18.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.397374848s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8202024-03-30T05:43:18.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 1.309149493s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8212024-03-30T05:43:18.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.157812974s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8222024-03-30T05:43:19.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 775.046184ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8232024-03-30T05:43:19.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 690.06206ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8242024-03-30T05:43:19.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.402281868s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8252024-03-30T05:43:20.433ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 568.225479ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8262024-03-30T05:43:20.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 779.229634ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8272024-03-30T05:43:21.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.377831532s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8282024-03-30T05:43:21.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 1.273797986s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8292024-03-30T05:43:21.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 837.841395ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8302024-03-30T05:43:22.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 594.671559ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8312024-03-30T05:43:22.398ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 594.915466ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8322024-03-30T05:43:22.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 852.9651ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8332024-03-30T05:43:22.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.099097027s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8342024-03-30T05:43:23.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 987.15692ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8352024-03-30T05:43:23.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 905.94825ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8362024-03-30T05:43:23.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f") failed: Property not found. retry in 1.457672751s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8372024-03-30T05:43:23.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 813.025857ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8382024-03-30T05:43:24.327ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465") failed: Property not found. retry in 634.542485ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8392024-03-30T05:43:24.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306") failed: Property not found. retry in 1.328548039s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8402024-03-30T05:43:26.171ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_cockroachdb_f7311d36-617f-4f72-be7a-3427dc69c306
8412024-03-30T05:43:26.171ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_cockroachdb_f7108ea5-965b-4e70-a9a0-7ffd5d13885f
8422024-03-30T05:43:26.171ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465
8432024-03-30T05:43:26.171ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_cockroachdb_f597bf55-7c6a-456c-8e41-286fd5037349
8442024-03-30T05:43:26.171ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_cockroachdb_ddf7c878-622c-4bc8-8b0b-5a158ff3b198
8452024-03-30T05:43:26.171ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8462024-03-30T05:43:26.172ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8472024-03-30T05:43:26.172ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 24012181 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:48593 req_id = 24bcb46e-feca-4962-9b5a-57786ee3c9cb response_code = 204 uri = /omicron-zones
8482024-03-30T05:43:26.206ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52474
8492024-03-30T05:43:26.236ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::6]:32221 file = sled-agent/src/services.rs:3408
8502024-03-30T05:43:35.453ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3426
8512024-03-30T05:43:36.976ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize CockroachDb: Error running command in zone 'oxz_cockroachdb_9d0139eb-7e07-43fa-92d7-f8a6676bf465': Command [/opt/oxide/cockroachdb/bin/cockroach sql --insecure --host [fd00:1122:3344:101::6]:32221 --file /opt/oxide/cockroachdb/sql/dbinit.sql] executed and failed with status: exit status: 1 stdout: BEGIN\nCREATE DATABASE\nCREATE ROLE\nALTER DEFAULT PRIVILEGES\nCONFIGURE ZONE 1\nCREATE TABLE\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TYPE\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE TABLE\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE INDEX\nCREATE TABLE\nCREATE TABLE\nCREATE VIEW\nCREATE TABLE\nCREATE INDEX\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE VIEW\nCREATE TYPE\nCREATE TABLE\nCREATE INDEX\nCREATE INDEX\nCREATE INDEX\nCREATE TABLE\nCREATE VIEW\nCREATE VIEW\nCREATE INDEX\nCREATE INDEX\nCREATE TYPE\n stderr: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=dbb6cfe3 key=/NamespaceTable/30/1/106/0/"public"/4/1 pri=0.08421992 epo=0 ts=1711777416.875419487,1 min=1711777416.375517229,0 seq=702} lock=true stat=ABORTED rts=1711777416.375517229,0 wto=false gul=1711777416.875517229,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_ABORT_SPAN): "sql txn" meta={id=dbb6cfe3 key=/NamespaceTable/30/1/106/0/"public"/4/1 pri=0.08421992 epo=0 ts=1711777416.875419487,1 min=1711777416.375517229,0 seq=702} lock=true stat=ABORTED rts=1711777416.375517229,0 wto=false gul=1711777416.875517229,0\nSQLSTATE: 40001\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html#abort_reason_abort_span\nFailed running "sql"\n file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 10769600 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:52474 req_id = 23af3898-6dae-4aea-977f-32b6ec174f42 response_code = 500 uri = /cockroachdb
8522024-03-30T05:43:36.984ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "23af3898-6dae-4aea-977f-32b6ec174f42", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "23af3898-6dae-4aea-977f-32b6ec174f42" } file = sled-agent/src/rack_setup/service.rs:952 retry_after = 77.403016ms
8532024-03-30T05:43:37.128ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::6]:32221 file = sled-agent/src/services.rs:3408
8542024-03-30T05:43:37.286ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3426
8552024-03-30T05:43:40.875ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3449
8562024-03-30T05:43:40.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3811892 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:52474 req_id = 5cde89ed-9b00-46a6-b7fb-ebc7181a4d96 response_code = 204 uri = /cockroachdb
8572024-03-30T05:43:40.940ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
8582024-03-30T05:43:40.941ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52853
8592024-03-30T05:43:40.944ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8602024-03-30T05:43:40.944ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
8612024-03-30T05:43:40.970ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8622024-03-30T05:43:40.970ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
8632024-03-30T05:43:40.989ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8642024-03-30T05:43:40.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
8652024-03-30T05:43:41.009ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8662024-03-30T05:43:41.009ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
8672024-03-30T05:43:41.029ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8682024-03-30T05:43:41.029ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
8692024-03-30T05:43:41.048ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8702024-03-30T05:43:41.048ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
8712024-03-30T05:43:41.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8722024-03-30T05:43:41.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
8732024-03-30T05:43:41.088ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8742024-03-30T05:43:41.088ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
8752024-03-30T05:43:41.108ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8762024-03-30T05:43:41.108ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
8772024-03-30T05:43:41.182ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8782024-03-30T05:43:41.182ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
8792024-03-30T05:43:41.262ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
8802024-03-30T05:43:41.262ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
8812024-03-30T05:43:41.337ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8822024-03-30T05:43:41.337ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
8832024-03-30T05:43:41.416ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8842024-03-30T05:43:41.416ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
8852024-03-30T05:43:41.487ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8862024-03-30T05:43:41.487ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
8872024-03-30T05:43:41.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8882024-03-30T05:43:41.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
8892024-03-30T05:43:41.651ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8902024-03-30T05:43:41.651ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
8912024-03-30T05:43:41.728ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8922024-03-30T05:43:41.728ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
8932024-03-30T05:43:41.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8942024-03-30T05:43:41.797ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
8952024-03-30T05:43:41.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8962024-03-30T05:43:41.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
8972024-03-30T05:43:41.921ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8982024-03-30T05:43:41.921ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
8992024-03-30T05:43:41.984ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9002024-03-30T05:43:41.984ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
9012024-03-30T05:43:41.985ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
9022024-03-30T05:43:41.985ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9032024-03-30T05:43:42.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
9042024-03-30T05:43:42.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
9052024-03-30T05:43:42.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
9062024-03-30T05:43:42.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
9072024-03-30T05:43:42.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
9082024-03-30T05:43:42.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
9092024-03-30T05:43:42.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
9102024-03-30T05:43:42.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
9112024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
9122024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
9132024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
9142024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
9152024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
9162024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
9172024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
9182024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
9192024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
9202024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
9212024-03-30T05:43:42.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
9222024-03-30T05:43:42.105ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 1163279 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 2d4b2754-f3de-45ba-b2f2-9dbfe727611b response_code = 500 uri = /omicron-zones
9232024-03-30T05:43:42.105ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2d4b2754-f3de-45ba-b2f2-9dbfe727611b", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:42 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2d4b2754-f3de-45ba-b2f2-9dbfe727611b" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 134.065463ms sled_address = [fd00:1122:3344:101::1]:12345
9242024-03-30T05:43:42.241ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
9252024-03-30T05:43:42.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9262024-03-30T05:43:42.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
9272024-03-30T05:43:42.263ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9282024-03-30T05:43:42.263ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
9292024-03-30T05:43:42.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9302024-03-30T05:43:42.285ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
9312024-03-30T05:43:42.305ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9322024-03-30T05:43:42.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
9332024-03-30T05:43:42.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9342024-03-30T05:43:42.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
9352024-03-30T05:43:42.345ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9362024-03-30T05:43:42.345ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
9372024-03-30T05:43:42.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9382024-03-30T05:43:42.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
9392024-03-30T05:43:42.386ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9402024-03-30T05:43:42.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
9412024-03-30T05:43:42.405ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9422024-03-30T05:43:42.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
9432024-03-30T05:43:42.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9442024-03-30T05:43:42.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
9452024-03-30T05:43:42.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
9462024-03-30T05:43:42.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
9472024-03-30T05:43:42.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9482024-03-30T05:43:42.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
9492024-03-30T05:43:42.486ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9502024-03-30T05:43:42.486ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
9512024-03-30T05:43:42.506ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9522024-03-30T05:43:42.506ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
9532024-03-30T05:43:42.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9542024-03-30T05:43:42.525ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
9552024-03-30T05:43:42.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9562024-03-30T05:43:42.544ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
9572024-03-30T05:43:42.564ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9582024-03-30T05:43:42.564ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
9592024-03-30T05:43:42.584ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9602024-03-30T05:43:42.584ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
9612024-03-30T05:43:42.588ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63548
9622024-03-30T05:43:42.589ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 179 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63548 req_id = 9c900e18-88fc-4d89-9878-a2a66fee50be response_code = 200 uri = /baseboard
9632024-03-30T05:43:42.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9642024-03-30T05:43:42.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
9652024-03-30T05:43:42.622ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9662024-03-30T05:43:42.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
9672024-03-30T05:43:42.642ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9682024-03-30T05:43:42.642ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
9692024-03-30T05:43:42.642ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
9702024-03-30T05:43:42.642ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9712024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
9722024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
9732024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
9742024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
9752024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
9762024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
9772024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
9782024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
9792024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
9802024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
9812024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
9822024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
9832024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
9842024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
9852024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
9862024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
9872024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
9882024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
9892024-03-30T05:43:42.737ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
9902024-03-30T05:43:42.737ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 495347 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 299aaf71-e852-416f-a85e-303d9387bce9 response_code = 500 uri = /omicron-zones
9912024-03-30T05:43:42.737ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "299aaf71-e852-416f-a85e-303d9387bce9", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:42 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "299aaf71-e852-416f-a85e-303d9387bce9" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 165.217808ms sled_address = [fd00:1122:3344:101::1]:12345
9922024-03-30T05:43:42.904ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
9932024-03-30T05:43:42.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9942024-03-30T05:43:42.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
9952024-03-30T05:43:42.925ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9962024-03-30T05:43:42.925ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
9972024-03-30T05:43:42.944ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9982024-03-30T05:43:42.944ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
9992024-03-30T05:43:42.960ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10002024-03-30T05:43:42.960ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
10012024-03-30T05:43:42.980ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10022024-03-30T05:43:42.980ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
10032024-03-30T05:43:42.997ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10042024-03-30T05:43:42.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
10052024-03-30T05:43:43.015ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10062024-03-30T05:43:43.015ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
10072024-03-30T05:43:43.033ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10082024-03-30T05:43:43.033ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
10092024-03-30T05:43:43.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10102024-03-30T05:43:43.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
10112024-03-30T05:43:43.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10122024-03-30T05:43:43.068ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
10132024-03-30T05:43:43.085ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
10142024-03-30T05:43:43.085ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
10152024-03-30T05:43:43.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10162024-03-30T05:43:43.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
10172024-03-30T05:43:43.119ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10182024-03-30T05:43:43.119ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
10192024-03-30T05:43:43.136ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10202024-03-30T05:43:43.136ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
10212024-03-30T05:43:43.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10222024-03-30T05:43:43.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
10232024-03-30T05:43:43.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10242024-03-30T05:43:43.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
10252024-03-30T05:43:43.187ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10262024-03-30T05:43:43.187ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
10272024-03-30T05:43:43.204ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10282024-03-30T05:43:43.204ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
10292024-03-30T05:43:43.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10302024-03-30T05:43:43.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
10312024-03-30T05:43:43.241ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10322024-03-30T05:43:43.241ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
10332024-03-30T05:43:43.258ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10342024-03-30T05:43:43.258ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
10352024-03-30T05:43:43.258ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
10362024-03-30T05:43:43.259ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10372024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
10382024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
10392024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
10402024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
10412024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
10422024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
10432024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
10442024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
10452024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
10462024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
10472024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
10482024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
10492024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
10502024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
10512024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
10522024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
10532024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
10542024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
10552024-03-30T05:43:43.346ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
10562024-03-30T05:43:43.346ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 442013 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 18086550-6dfa-40bc-9369-7900ce051b76 response_code = 500 uri = /omicron-zones
10572024-03-30T05:43:43.346ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "18086550-6dfa-40bc-9369-7900ce051b76", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "18086550-6dfa-40bc-9369-7900ce051b76" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 201.729238ms sled_address = [fd00:1122:3344:101::1]:12345
10582024-03-30T05:43:43.550ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
10592024-03-30T05:43:43.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10602024-03-30T05:43:43.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
10612024-03-30T05:43:43.571ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10622024-03-30T05:43:43.571ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
10632024-03-30T05:43:43.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10642024-03-30T05:43:43.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
10652024-03-30T05:43:43.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10662024-03-30T05:43:43.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
10672024-03-30T05:43:43.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10682024-03-30T05:43:43.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
10692024-03-30T05:43:43.650ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10702024-03-30T05:43:43.650ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
10712024-03-30T05:43:43.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10722024-03-30T05:43:43.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
10732024-03-30T05:43:43.688ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10742024-03-30T05:43:43.688ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
10752024-03-30T05:43:43.706ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10762024-03-30T05:43:43.706ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
10772024-03-30T05:43:43.724ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10782024-03-30T05:43:43.724ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
10792024-03-30T05:43:43.742ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
10802024-03-30T05:43:43.742ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
10812024-03-30T05:43:43.759ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10822024-03-30T05:43:43.759ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
10832024-03-30T05:43:43.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10842024-03-30T05:43:43.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
10852024-03-30T05:43:43.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10862024-03-30T05:43:43.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
10872024-03-30T05:43:43.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10882024-03-30T05:43:43.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
10892024-03-30T05:43:43.830ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10902024-03-30T05:43:43.830ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
10912024-03-30T05:43:43.848ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10922024-03-30T05:43:43.848ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
10932024-03-30T05:43:43.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10942024-03-30T05:43:43.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
10952024-03-30T05:43:43.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10962024-03-30T05:43:43.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
10972024-03-30T05:43:43.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10982024-03-30T05:43:43.901ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
10992024-03-30T05:43:43.920ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11002024-03-30T05:43:43.920ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
11012024-03-30T05:43:43.920ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
11022024-03-30T05:43:43.920ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11032024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
11042024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
11052024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
11062024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
11072024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
11082024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
11092024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
11102024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
11112024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
11122024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
11132024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
11142024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
11152024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
11162024-03-30T05:43:44.007ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
11172024-03-30T05:43:44.008ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
11182024-03-30T05:43:44.008ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
11192024-03-30T05:43:44.008ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
11202024-03-30T05:43:44.008ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
11212024-03-30T05:43:44.008ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
11222024-03-30T05:43:44.008ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 457578 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = dca4100b-b0bf-433b-8b37-d40458fd862c response_code = 500 uri = /omicron-zones
11232024-03-30T05:43:44.008ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dca4100b-b0bf-433b-8b37-d40458fd862c", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dca4100b-b0bf-433b-8b37-d40458fd862c" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 242.040673ms sled_address = [fd00:1122:3344:101::1]:12345
11242024-03-30T05:43:44.252ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
11252024-03-30T05:43:44.252ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11262024-03-30T05:43:44.252ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
11272024-03-30T05:43:44.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11282024-03-30T05:43:44.274ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
11292024-03-30T05:43:44.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11302024-03-30T05:43:44.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
11312024-03-30T05:43:44.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11322024-03-30T05:43:44.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
11332024-03-30T05:43:44.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11342024-03-30T05:43:44.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
11352024-03-30T05:43:44.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11362024-03-30T05:43:44.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
11372024-03-30T05:43:44.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11382024-03-30T05:43:44.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
11392024-03-30T05:43:44.384ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11402024-03-30T05:43:44.384ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
11412024-03-30T05:43:44.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11422024-03-30T05:43:44.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
11432024-03-30T05:43:44.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11442024-03-30T05:43:44.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
11452024-03-30T05:43:44.437ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
11462024-03-30T05:43:44.437ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
11472024-03-30T05:43:44.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11482024-03-30T05:43:44.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
11492024-03-30T05:43:44.472ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11502024-03-30T05:43:44.472ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
11512024-03-30T05:43:44.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11522024-03-30T05:43:44.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
11532024-03-30T05:43:44.510ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11542024-03-30T05:43:44.510ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
11552024-03-30T05:43:44.530ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11562024-03-30T05:43:44.530ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
11572024-03-30T05:43:44.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11582024-03-30T05:43:44.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
11592024-03-30T05:43:44.569ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11602024-03-30T05:43:44.569ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
11612024-03-30T05:43:44.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11622024-03-30T05:43:44.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
11632024-03-30T05:43:44.609ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11642024-03-30T05:43:44.609ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
11652024-03-30T05:43:44.629ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11662024-03-30T05:43:44.629ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
11672024-03-30T05:43:44.629ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
11682024-03-30T05:43:44.629ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11692024-03-30T05:43:44.718ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
11702024-03-30T05:43:44.718ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
11712024-03-30T05:43:44.718ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
11722024-03-30T05:43:44.718ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
11732024-03-30T05:43:44.718ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
11742024-03-30T05:43:44.718ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
11752024-03-30T05:43:44.718ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
11762024-03-30T05:43:44.718ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
11772024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
11782024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
11792024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
11802024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
11812024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
11822024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
11832024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
11842024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
11852024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
11862024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
11872024-03-30T05:43:44.719ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
11882024-03-30T05:43:44.719ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 466587 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 9e90e225-17ed-4109-aeb9-4c6371ab595a response_code = 500 uri = /omicron-zones
11892024-03-30T05:43:44.719ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9e90e225-17ed-4109-aeb9-4c6371ab595a", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9e90e225-17ed-4109-aeb9-4c6371ab595a" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 177.26639ms sled_address = [fd00:1122:3344:101::1]:12345
11902024-03-30T05:43:44.897ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
11912024-03-30T05:43:44.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11922024-03-30T05:43:44.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
11932024-03-30T05:43:44.918ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11942024-03-30T05:43:44.918ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
11952024-03-30T05:43:44.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11962024-03-30T05:43:44.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
11972024-03-30T05:43:44.955ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11982024-03-30T05:43:44.955ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
11992024-03-30T05:43:44.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12002024-03-30T05:43:44.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
12012024-03-30T05:43:44.989ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12022024-03-30T05:43:44.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
12032024-03-30T05:43:45.007ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12042024-03-30T05:43:45.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
12052024-03-30T05:43:45.026ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12062024-03-30T05:43:45.026ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
12072024-03-30T05:43:45.044ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12082024-03-30T05:43:45.044ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
12092024-03-30T05:43:45.062ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12102024-03-30T05:43:45.062ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
12112024-03-30T05:43:45.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
12122024-03-30T05:43:45.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
12132024-03-30T05:43:45.098ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12142024-03-30T05:43:45.098ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
12152024-03-30T05:43:45.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12162024-03-30T05:43:45.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
12172024-03-30T05:43:45.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12182024-03-30T05:43:45.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
12192024-03-30T05:43:45.150ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12202024-03-30T05:43:45.150ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
12212024-03-30T05:43:45.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12222024-03-30T05:43:45.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
12232024-03-30T05:43:45.186ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12242024-03-30T05:43:45.186ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
12252024-03-30T05:43:45.205ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12262024-03-30T05:43:45.205ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
12272024-03-30T05:43:45.222ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12282024-03-30T05:43:45.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
12292024-03-30T05:43:45.240ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12302024-03-30T05:43:45.240ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
12312024-03-30T05:43:45.257ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12322024-03-30T05:43:45.257ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
12332024-03-30T05:43:45.257ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
12342024-03-30T05:43:45.258ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12352024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
12362024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
12372024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
12382024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
12392024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
12402024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
12412024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
12422024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
12432024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
12442024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
12452024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
12462024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
12472024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
12482024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
12492024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
12502024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
12512024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
12522024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
12532024-03-30T05:43:45.353ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
12542024-03-30T05:43:45.354ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 455713 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 74d41e00-0b2d-48ef-be6a-7bff5405f9c6 response_code = 500 uri = /omicron-zones
12552024-03-30T05:43:45.354ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "74d41e00-0b2d-48ef-be6a-7bff5405f9c6", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:44 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "74d41e00-0b2d-48ef-be6a-7bff5405f9c6" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 156.421207ms sled_address = [fd00:1122:3344:101::1]:12345
12562024-03-30T05:43:45.512ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
12572024-03-30T05:43:45.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12582024-03-30T05:43:45.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
12592024-03-30T05:43:45.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12602024-03-30T05:43:45.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
12612024-03-30T05:43:45.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12622024-03-30T05:43:45.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
12632024-03-30T05:43:45.575ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12642024-03-30T05:43:45.575ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
12652024-03-30T05:43:45.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12662024-03-30T05:43:45.595ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
12672024-03-30T05:43:45.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12682024-03-30T05:43:45.615ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
12692024-03-30T05:43:45.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12702024-03-30T05:43:45.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
12712024-03-30T05:43:45.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12722024-03-30T05:43:45.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
12732024-03-30T05:43:45.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12742024-03-30T05:43:45.680ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
12752024-03-30T05:43:45.703ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12762024-03-30T05:43:45.703ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
12772024-03-30T05:43:45.726ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
12782024-03-30T05:43:45.726ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
12792024-03-30T05:43:45.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12802024-03-30T05:43:45.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
12812024-03-30T05:43:45.771ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12822024-03-30T05:43:45.771ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
12832024-03-30T05:43:45.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12842024-03-30T05:43:45.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
12852024-03-30T05:43:45.815ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12862024-03-30T05:43:45.815ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
12872024-03-30T05:43:45.836ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12882024-03-30T05:43:45.836ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
12892024-03-30T05:43:45.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12902024-03-30T05:43:45.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
12912024-03-30T05:43:45.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12922024-03-30T05:43:45.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
12932024-03-30T05:43:45.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12942024-03-30T05:43:45.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
12952024-03-30T05:43:45.922ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12962024-03-30T05:43:45.922ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
12972024-03-30T05:43:45.943ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12982024-03-30T05:43:45.943ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
12992024-03-30T05:43:45.943ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
13002024-03-30T05:43:45.943ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13012024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
13022024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
13032024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
13042024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
13052024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
13062024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
13072024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
13082024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
13092024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
13102024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
13112024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
13122024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
13132024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
13142024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
13152024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
13162024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
13172024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
13182024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
13192024-03-30T05:43:46.044ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
13202024-03-30T05:43:46.044ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 531802 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 76d0150b-21e1-44b1-a704-4154d72dcb84 response_code = 500 uri = /omicron-zones
13212024-03-30T05:43:46.044ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "76d0150b-21e1-44b1-a704-4154d72dcb84", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "76d0150b-21e1-44b1-a704-4154d72dcb84" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 384.158462ms sled_address = [fd00:1122:3344:101::1]:12345
13222024-03-30T05:43:46.430ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
13232024-03-30T05:43:46.431ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13242024-03-30T05:43:46.431ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
13252024-03-30T05:43:46.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13262024-03-30T05:43:46.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
13272024-03-30T05:43:46.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13282024-03-30T05:43:46.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
13292024-03-30T05:43:46.500ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13302024-03-30T05:43:46.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
13312024-03-30T05:43:46.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13322024-03-30T05:43:46.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
13332024-03-30T05:43:46.543ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13342024-03-30T05:43:46.543ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
13352024-03-30T05:43:46.564ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13362024-03-30T05:43:46.564ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
13372024-03-30T05:43:46.586ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13382024-03-30T05:43:46.586ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
13392024-03-30T05:43:46.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13402024-03-30T05:43:46.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
13412024-03-30T05:43:46.631ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13422024-03-30T05:43:46.631ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
13432024-03-30T05:43:46.655ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
13442024-03-30T05:43:46.655ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
13452024-03-30T05:43:46.677ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13462024-03-30T05:43:46.677ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
13472024-03-30T05:43:46.697ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13482024-03-30T05:43:46.697ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
13492024-03-30T05:43:46.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13502024-03-30T05:43:46.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
13512024-03-30T05:43:46.739ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13522024-03-30T05:43:46.739ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
13532024-03-30T05:43:46.761ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13542024-03-30T05:43:46.761ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
13552024-03-30T05:43:46.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13562024-03-30T05:43:46.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
13572024-03-30T05:43:46.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13582024-03-30T05:43:46.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
13592024-03-30T05:43:46.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13602024-03-30T05:43:46.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
13612024-03-30T05:43:46.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13622024-03-30T05:43:46.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
13632024-03-30T05:43:46.866ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13642024-03-30T05:43:46.866ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
13652024-03-30T05:43:46.866ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
13662024-03-30T05:43:46.867ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13672024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
13682024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
13692024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
13702024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
13712024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
13722024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
13732024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
13742024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
13752024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
13762024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
13772024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
13782024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
13792024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
13802024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
13812024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
13822024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
13832024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
13842024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
13852024-03-30T05:43:46.967ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
13862024-03-30T05:43:46.968ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 537477 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = bab56f20-9f85-4379-8450-24230bd5449e response_code = 500 uri = /omicron-zones
13872024-03-30T05:43:46.968ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bab56f20-9f85-4379-8450-24230bd5449e", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:46 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bab56f20-9f85-4379-8450-24230bd5449e" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 318.517218ms sled_address = [fd00:1122:3344:101::1]:12345
13882024-03-30T05:43:47.287ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
13892024-03-30T05:43:47.288ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13902024-03-30T05:43:47.288ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
13912024-03-30T05:43:47.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13922024-03-30T05:43:47.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
13932024-03-30T05:43:47.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13942024-03-30T05:43:47.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
13952024-03-30T05:43:47.352ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13962024-03-30T05:43:47.352ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
13972024-03-30T05:43:47.372ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13982024-03-30T05:43:47.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
13992024-03-30T05:43:47.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14002024-03-30T05:43:47.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
14012024-03-30T05:43:47.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14022024-03-30T05:43:47.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
14032024-03-30T05:43:47.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14042024-03-30T05:43:47.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
14052024-03-30T05:43:47.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14062024-03-30T05:43:47.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
14072024-03-30T05:43:47.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14082024-03-30T05:43:47.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
14092024-03-30T05:43:47.484ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
14102024-03-30T05:43:47.484ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
14112024-03-30T05:43:47.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14122024-03-30T05:43:47.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
14132024-03-30T05:43:47.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14142024-03-30T05:43:47.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
14152024-03-30T05:43:47.540ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14162024-03-30T05:43:47.540ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
14172024-03-30T05:43:47.558ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14182024-03-30T05:43:47.558ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
14192024-03-30T05:43:47.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14202024-03-30T05:43:47.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
14212024-03-30T05:43:47.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14222024-03-30T05:43:47.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
14232024-03-30T05:43:47.617ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14242024-03-30T05:43:47.617ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
14252024-03-30T05:43:47.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14262024-03-30T05:43:47.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
14272024-03-30T05:43:47.653ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14282024-03-30T05:43:47.653ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
14292024-03-30T05:43:47.671ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14302024-03-30T05:43:47.671ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
14312024-03-30T05:43:47.672ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
14322024-03-30T05:43:47.672ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14332024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
14342024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
14352024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
14362024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
14372024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
14382024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
14392024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
14402024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
14412024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
14422024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
14432024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
14442024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
14452024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
14462024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
14472024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
14482024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
14492024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
14502024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
14512024-03-30T05:43:47.765ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
14522024-03-30T05:43:47.765ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 477129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 5648a655-55d8-4eca-b45a-49def89502eb response_code = 500 uri = /omicron-zones
14532024-03-30T05:43:47.765ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5648a655-55d8-4eca-b45a-49def89502eb", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5648a655-55d8-4eca-b45a-49def89502eb" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 572.229538ms sled_address = [fd00:1122:3344:101::1]:12345
14542024-03-30T05:43:48.338ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
14552024-03-30T05:43:48.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14562024-03-30T05:43:48.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
14572024-03-30T05:43:48.360ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14582024-03-30T05:43:48.360ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
14592024-03-30T05:43:48.379ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14602024-03-30T05:43:48.379ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
14612024-03-30T05:43:48.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14622024-03-30T05:43:48.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
14632024-03-30T05:43:48.422ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14642024-03-30T05:43:48.422ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
14652024-03-30T05:43:48.443ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14662024-03-30T05:43:48.443ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
14672024-03-30T05:43:48.463ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14682024-03-30T05:43:48.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
14692024-03-30T05:43:48.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14702024-03-30T05:43:48.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
14712024-03-30T05:43:48.504ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14722024-03-30T05:43:48.504ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
14732024-03-30T05:43:48.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14742024-03-30T05:43:48.525ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
14752024-03-30T05:43:48.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
14762024-03-30T05:43:48.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
14772024-03-30T05:43:48.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14782024-03-30T05:43:48.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
14792024-03-30T05:43:48.591ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14802024-03-30T05:43:48.591ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
14812024-03-30T05:43:48.612ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14822024-03-30T05:43:48.612ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
14832024-03-30T05:43:48.634ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14842024-03-30T05:43:48.634ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
14852024-03-30T05:43:48.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14862024-03-30T05:43:48.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
14872024-03-30T05:43:48.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14882024-03-30T05:43:48.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
14892024-03-30T05:43:48.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14902024-03-30T05:43:48.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
14912024-03-30T05:43:48.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14922024-03-30T05:43:48.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
14932024-03-30T05:43:48.744ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14942024-03-30T05:43:48.744ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
14952024-03-30T05:43:48.765ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14962024-03-30T05:43:48.765ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
14972024-03-30T05:43:48.765ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
14982024-03-30T05:43:48.765ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14992024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
15002024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
15012024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
15022024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
15032024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
15042024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
15052024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
15062024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
15072024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
15082024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
15092024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
15102024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
15112024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
15122024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
15132024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
15142024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
15152024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
15162024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
15172024-03-30T05:43:48.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
15182024-03-30T05:43:48.864ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 524897 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = a93892da-5a22-4ac4-adfb-0e1abf2ea0d1 response_code = 500 uri = /omicron-zones
15192024-03-30T05:43:48.864ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a93892da-5a22-4ac4-adfb-0e1abf2ea0d1", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a93892da-5a22-4ac4-adfb-0e1abf2ea0d1" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 269.832933ms sled_address = [fd00:1122:3344:101::1]:12345
15202024-03-30T05:43:49.134ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
15212024-03-30T05:43:49.135ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15222024-03-30T05:43:49.135ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
15232024-03-30T05:43:49.157ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15242024-03-30T05:43:49.157ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
15252024-03-30T05:43:49.176ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15262024-03-30T05:43:49.176ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
15272024-03-30T05:43:49.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15282024-03-30T05:43:49.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
15292024-03-30T05:43:49.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15302024-03-30T05:43:49.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
15312024-03-30T05:43:49.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15322024-03-30T05:43:49.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
15332024-03-30T05:43:49.248ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15342024-03-30T05:43:49.248ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
15352024-03-30T05:43:49.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15362024-03-30T05:43:49.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
15372024-03-30T05:43:49.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15382024-03-30T05:43:49.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
15392024-03-30T05:43:49.305ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15402024-03-30T05:43:49.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
15412024-03-30T05:43:49.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
15422024-03-30T05:43:49.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
15432024-03-30T05:43:49.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15442024-03-30T05:43:49.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
15452024-03-30T05:43:49.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15462024-03-30T05:43:49.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
15472024-03-30T05:43:49.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15482024-03-30T05:43:49.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
15492024-03-30T05:43:49.404ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15502024-03-30T05:43:49.404ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
15512024-03-30T05:43:49.422ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15522024-03-30T05:43:49.422ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
15532024-03-30T05:43:49.440ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15542024-03-30T05:43:49.440ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
15552024-03-30T05:43:49.461ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15562024-03-30T05:43:49.461ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
15572024-03-30T05:43:49.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15582024-03-30T05:43:49.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
15592024-03-30T05:43:49.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15602024-03-30T05:43:49.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
15612024-03-30T05:43:49.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15622024-03-30T05:43:49.520ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
15632024-03-30T05:43:49.521ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
15642024-03-30T05:43:49.521ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15652024-03-30T05:43:49.620ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
15662024-03-30T05:43:49.620ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
15672024-03-30T05:43:49.620ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
15682024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
15692024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
15702024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
15712024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
15722024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
15732024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
15742024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
15752024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
15762024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
15772024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
15782024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
15792024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
15802024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
15812024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
15822024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
15832024-03-30T05:43:49.621ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
15842024-03-30T05:43:49.621ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 486166 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 53c74a00-1dba-46ba-b84f-612e0f4f5094 response_code = 500 uri = /omicron-zones
15852024-03-30T05:43:49.621ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "53c74a00-1dba-46ba-b84f-612e0f4f5094", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "53c74a00-1dba-46ba-b84f-612e0f4f5094" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 502.093187ms sled_address = [fd00:1122:3344:101::1]:12345
15862024-03-30T05:43:50.124ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
15872024-03-30T05:43:50.125ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15882024-03-30T05:43:50.125ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
15892024-03-30T05:43:50.148ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15902024-03-30T05:43:50.148ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
15912024-03-30T05:43:50.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15922024-03-30T05:43:50.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
15932024-03-30T05:43:50.187ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15942024-03-30T05:43:50.187ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
15952024-03-30T05:43:50.206ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15962024-03-30T05:43:50.206ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
15972024-03-30T05:43:50.227ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15982024-03-30T05:43:50.227ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
15992024-03-30T05:43:50.246ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16002024-03-30T05:43:50.247ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
16012024-03-30T05:43:50.266ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16022024-03-30T05:43:50.266ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
16032024-03-30T05:43:50.286ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16042024-03-30T05:43:50.286ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
16052024-03-30T05:43:50.307ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16062024-03-30T05:43:50.307ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
16072024-03-30T05:43:50.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
16082024-03-30T05:43:50.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
16092024-03-30T05:43:50.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16102024-03-30T05:43:50.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
16112024-03-30T05:43:50.369ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16122024-03-30T05:43:50.369ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
16132024-03-30T05:43:50.389ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16142024-03-30T05:43:50.389ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
16152024-03-30T05:43:50.410ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16162024-03-30T05:43:50.410ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
16172024-03-30T05:43:50.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16182024-03-30T05:43:50.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
16192024-03-30T05:43:50.450ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16202024-03-30T05:43:50.450ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
16212024-03-30T05:43:50.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16222024-03-30T05:43:50.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
16232024-03-30T05:43:50.490ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16242024-03-30T05:43:50.490ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
16252024-03-30T05:43:50.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16262024-03-30T05:43:50.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
16272024-03-30T05:43:50.528ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16282024-03-30T05:43:50.528ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
16292024-03-30T05:43:50.529ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
16302024-03-30T05:43:50.529ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16312024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
16322024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
16332024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
16342024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
16352024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
16362024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
16372024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
16382024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
16392024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
16402024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
16412024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
16422024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
16432024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
16442024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
16452024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
16462024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
16472024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
16482024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
16492024-03-30T05:43:50.624ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
16502024-03-30T05:43:50.625ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 500003 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = d0b16590-96ff-4292-9400-f83aaee4626f response_code = 500 uri = /omicron-zones
16512024-03-30T05:43:50.625ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d0b16590-96ff-4292-9400-f83aaee4626f", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:50 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d0b16590-96ff-4292-9400-f83aaee4626f" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 492.655909ms sled_address = [fd00:1122:3344:101::1]:12345
16522024-03-30T05:43:51.118ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
16532024-03-30T05:43:51.119ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16542024-03-30T05:43:51.119ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
16552024-03-30T05:43:51.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16562024-03-30T05:43:51.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
16572024-03-30T05:43:51.162ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16582024-03-30T05:43:51.162ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
16592024-03-30T05:43:51.181ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16602024-03-30T05:43:51.181ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
16612024-03-30T05:43:51.201ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16622024-03-30T05:43:51.201ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
16632024-03-30T05:43:51.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16642024-03-30T05:43:51.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
16652024-03-30T05:43:51.241ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16662024-03-30T05:43:51.241ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
16672024-03-30T05:43:51.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16682024-03-30T05:43:51.261ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
16692024-03-30T05:43:51.281ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16702024-03-30T05:43:51.281ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
16712024-03-30T05:43:51.299ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16722024-03-30T05:43:51.299ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
16732024-03-30T05:43:51.318ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
16742024-03-30T05:43:51.318ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
16752024-03-30T05:43:51.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16762024-03-30T05:43:51.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
16772024-03-30T05:43:51.359ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16782024-03-30T05:43:51.359ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
16792024-03-30T05:43:51.382ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16802024-03-30T05:43:51.382ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
16812024-03-30T05:43:51.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16822024-03-30T05:43:51.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
16832024-03-30T05:43:51.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16842024-03-30T05:43:51.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
16852024-03-30T05:43:51.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16862024-03-30T05:43:51.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
16872024-03-30T05:43:51.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16882024-03-30T05:43:51.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
16892024-03-30T05:43:51.480ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16902024-03-30T05:43:51.480ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
16912024-03-30T05:43:51.500ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16922024-03-30T05:43:51.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
16932024-03-30T05:43:51.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16942024-03-30T05:43:51.520ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
16952024-03-30T05:43:51.520ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
16962024-03-30T05:43:51.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16972024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
16982024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
16992024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
17002024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
17012024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
17022024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
17032024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
17042024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
17052024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
17062024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
17072024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
17082024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
17092024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
17102024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
17112024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
17122024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
17132024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
17142024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
17152024-03-30T05:43:51.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
17162024-03-30T05:43:51.615ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 496849 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 71e088bc-0239-4d68-bd9c-7d33dc9720f8 response_code = 500 uri = /omicron-zones
17172024-03-30T05:43:51.616ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "71e088bc-0239-4d68-bd9c-7d33dc9720f8", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "71e088bc-0239-4d68-bd9c-7d33dc9720f8" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 673.639551ms sled_address = [fd00:1122:3344:101::1]:12345
17182024-03-30T05:43:52.291ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
17192024-03-30T05:43:52.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17202024-03-30T05:43:52.291ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
17212024-03-30T05:43:52.314ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17222024-03-30T05:43:52.314ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
17232024-03-30T05:43:52.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17242024-03-30T05:43:52.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
17252024-03-30T05:43:52.354ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17262024-03-30T05:43:52.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
17272024-03-30T05:43:52.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17282024-03-30T05:43:52.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
17292024-03-30T05:43:52.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17302024-03-30T05:43:52.394ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
17312024-03-30T05:43:52.413ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17322024-03-30T05:43:52.413ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
17332024-03-30T05:43:52.431ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17342024-03-30T05:43:52.431ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
17352024-03-30T05:43:52.449ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17362024-03-30T05:43:52.449ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
17372024-03-30T05:43:52.468ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17382024-03-30T05:43:52.468ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
17392024-03-30T05:43:52.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
17402024-03-30T05:43:52.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
17412024-03-30T05:43:52.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17422024-03-30T05:43:52.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
17432024-03-30T05:43:52.530ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17442024-03-30T05:43:52.530ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
17452024-03-30T05:43:52.548ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17462024-03-30T05:43:52.548ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
17472024-03-30T05:43:52.569ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17482024-03-30T05:43:52.569ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
17492024-03-30T05:43:52.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17502024-03-30T05:43:52.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
17512024-03-30T05:43:52.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17522024-03-30T05:43:52.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
17532024-03-30T05:43:52.623ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17542024-03-30T05:43:52.623ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
17552024-03-30T05:43:52.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17562024-03-30T05:43:52.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
17572024-03-30T05:43:52.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17582024-03-30T05:43:52.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
17592024-03-30T05:43:52.683ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17602024-03-30T05:43:52.683ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
17612024-03-30T05:43:52.683ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
17622024-03-30T05:43:52.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17632024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
17642024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
17652024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
17662024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
17672024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
17682024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
17692024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
17702024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
17712024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
17722024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
17732024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
17742024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
17752024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
17762024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
17772024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
17782024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
17792024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
17802024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
17812024-03-30T05:43:52.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
17822024-03-30T05:43:52.784ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 493554 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = a2fad64f-da37-483e-b393-7cdc82e953ac response_code = 500 uri = /omicron-zones
17832024-03-30T05:43:52.785ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a2fad64f-da37-483e-b393-7cdc82e953ac", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a2fad64f-da37-483e-b393-7cdc82e953ac" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 1.326579596s sled_address = [fd00:1122:3344:101::1]:12345
17842024-03-30T05:43:54.112ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
17852024-03-30T05:43:54.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17862024-03-30T05:43:54.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
17872024-03-30T05:43:54.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17882024-03-30T05:43:54.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
17892024-03-30T05:43:54.156ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17902024-03-30T05:43:54.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
17912024-03-30T05:43:54.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17922024-03-30T05:43:54.177ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
17932024-03-30T05:43:54.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17942024-03-30T05:43:54.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
17952024-03-30T05:43:54.212ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17962024-03-30T05:43:54.212ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
17972024-03-30T05:43:54.231ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17982024-03-30T05:43:54.231ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
17992024-03-30T05:43:54.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18002024-03-30T05:43:54.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
18012024-03-30T05:43:54.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18022024-03-30T05:43:54.271ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
18032024-03-30T05:43:54.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18042024-03-30T05:43:54.291ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
18052024-03-30T05:43:54.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
18062024-03-30T05:43:54.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
18072024-03-30T05:43:54.327ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18082024-03-30T05:43:54.327ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
18092024-03-30T05:43:54.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18102024-03-30T05:43:54.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
18112024-03-30T05:43:54.365ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18122024-03-30T05:43:54.365ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
18132024-03-30T05:43:54.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18142024-03-30T05:43:54.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
18152024-03-30T05:43:54.402ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18162024-03-30T05:43:54.402ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
18172024-03-30T05:43:54.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18182024-03-30T05:43:54.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
18192024-03-30T05:43:54.445ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18202024-03-30T05:43:54.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
18212024-03-30T05:43:54.466ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18222024-03-30T05:43:54.466ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
18232024-03-30T05:43:54.487ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18242024-03-30T05:43:54.487ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
18252024-03-30T05:43:54.509ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18262024-03-30T05:43:54.509ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
18272024-03-30T05:43:54.509ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
18282024-03-30T05:43:54.509ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18292024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
18302024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
18312024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
18322024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
18332024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
18342024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
18352024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
18362024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
18372024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
18382024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
18392024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
18402024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
18412024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
18422024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
18432024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
18442024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
18452024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
18462024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
18472024-03-30T05:43:54.609ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
18482024-03-30T05:43:54.609ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 496766 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 938cfe04-82d9-4813-b7d1-cc13615a62fc response_code = 500 uri = /omicron-zones
18492024-03-30T05:43:54.610ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "938cfe04-82d9-4813-b7d1-cc13615a62fc", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "938cfe04-82d9-4813-b7d1-cc13615a62fc" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 1.033950406s sled_address = [fd00:1122:3344:101::1]:12345
18502024-03-30T05:43:55.644ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
18512024-03-30T05:43:55.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18522024-03-30T05:43:55.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
18532024-03-30T05:43:55.668ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18542024-03-30T05:43:55.668ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
18552024-03-30T05:43:55.690ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18562024-03-30T05:43:55.690ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
18572024-03-30T05:43:55.711ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18582024-03-30T05:43:55.711ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
18592024-03-30T05:43:55.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18602024-03-30T05:43:55.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
18612024-03-30T05:43:55.751ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18622024-03-30T05:43:55.751ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
18632024-03-30T05:43:55.772ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18642024-03-30T05:43:55.772ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
18652024-03-30T05:43:55.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18662024-03-30T05:43:55.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
18672024-03-30T05:43:55.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18682024-03-30T05:43:55.811ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
18692024-03-30T05:43:55.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18702024-03-30T05:43:55.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
18712024-03-30T05:43:55.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
18722024-03-30T05:43:55.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
18732024-03-30T05:43:55.870ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18742024-03-30T05:43:55.870ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
18752024-03-30T05:43:55.889ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18762024-03-30T05:43:55.889ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
18772024-03-30T05:43:55.910ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18782024-03-30T05:43:55.910ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
18792024-03-30T05:43:55.931ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18802024-03-30T05:43:55.931ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
18812024-03-30T05:43:55.949ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18822024-03-30T05:43:55.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
18832024-03-30T05:43:55.970ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18842024-03-30T05:43:55.970ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
18852024-03-30T05:43:55.988ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18862024-03-30T05:43:55.989ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
18872024-03-30T05:43:56.009ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18882024-03-30T05:43:56.009ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
18892024-03-30T05:43:56.027ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18902024-03-30T05:43:56.027ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
18912024-03-30T05:43:56.047ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18922024-03-30T05:43:56.047ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
18932024-03-30T05:43:56.047ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
18942024-03-30T05:43:56.047ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18952024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
18962024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
18972024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
18982024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
18992024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
19002024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
19012024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
19022024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
19032024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
19042024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
19052024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
19062024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
19072024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
19082024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
19092024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
19102024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
19112024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
19122024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
19132024-03-30T05:43:56.138ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
19142024-03-30T05:43:56.138ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 493977 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 5d914cec-a688-416c-82b9-557e38538c33 response_code = 500 uri = /omicron-zones
19152024-03-30T05:43:56.139ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5d914cec-a688-416c-82b9-557e38538c33", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5d914cec-a688-416c-82b9-557e38538c33" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 2.10162743s sled_address = [fd00:1122:3344:101::1]:12345
19162024-03-30T05:43:58.242ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
19172024-03-30T05:43:58.243ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19182024-03-30T05:43:58.243ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
19192024-03-30T05:43:58.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19202024-03-30T05:43:58.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
19212024-03-30T05:43:58.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19222024-03-30T05:43:58.285ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
19232024-03-30T05:43:58.304ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19242024-03-30T05:43:58.304ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
19252024-03-30T05:43:58.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19262024-03-30T05:43:58.322ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
19272024-03-30T05:43:58.340ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19282024-03-30T05:43:58.340ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
19292024-03-30T05:43:58.358ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19302024-03-30T05:43:58.358ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
19312024-03-30T05:43:58.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19322024-03-30T05:43:58.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
19332024-03-30T05:43:58.393ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19342024-03-30T05:43:58.393ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
19352024-03-30T05:43:58.411ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19362024-03-30T05:43:58.411ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
19372024-03-30T05:43:58.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
19382024-03-30T05:43:58.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
19392024-03-30T05:43:58.449ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19402024-03-30T05:43:58.449ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
19412024-03-30T05:43:58.466ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19422024-03-30T05:43:58.466ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
19432024-03-30T05:43:58.485ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19442024-03-30T05:43:58.485ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
19452024-03-30T05:43:58.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19462024-03-30T05:43:58.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
19472024-03-30T05:43:58.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19482024-03-30T05:43:58.525ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
19492024-03-30T05:43:58.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19502024-03-30T05:43:58.544ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
19512024-03-30T05:43:58.564ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19522024-03-30T05:43:58.564ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
19532024-03-30T05:43:58.585ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19542024-03-30T05:43:58.585ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
19552024-03-30T05:43:58.605ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19562024-03-30T05:43:58.605ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
19572024-03-30T05:43:58.624ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19582024-03-30T05:43:58.624ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
19592024-03-30T05:43:58.624ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
19602024-03-30T05:43:58.624ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19612024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
19622024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
19632024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
19642024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
19652024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
19662024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
19672024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
19682024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
19692024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
19702024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
19712024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
19722024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
19732024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
19742024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
19752024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
19762024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
19772024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
19782024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
19792024-03-30T05:43:58.715ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
19802024-03-30T05:43:58.715ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 472952 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 8eddc4ca-05b8-416b-9ea7-28d8829a2592 response_code = 500 uri = /omicron-zones
19812024-03-30T05:43:58.715ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8eddc4ca-05b8-416b-9ea7-28d8829a2592", "content-length": "124", "date": "Sat, 30 Mar 2024 05:43:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8eddc4ca-05b8-416b-9ea7-28d8829a2592" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 1.20955001s sled_address = [fd00:1122:3344:101::1]:12345
19822024-03-30T05:43:59.926ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
19832024-03-30T05:43:59.927ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19842024-03-30T05:43:59.927ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
19852024-03-30T05:43:59.948ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19862024-03-30T05:43:59.948ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
19872024-03-30T05:43:59.968ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19882024-03-30T05:43:59.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
19892024-03-30T05:43:59.986ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19902024-03-30T05:43:59.986ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
19912024-03-30T05:44:00.004ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19922024-03-30T05:44:00.004ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
19932024-03-30T05:44:00.022ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19942024-03-30T05:44:00.022ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
19952024-03-30T05:44:00.040ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19962024-03-30T05:44:00.040ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
19972024-03-30T05:44:00.057ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19982024-03-30T05:44:00.057ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
19992024-03-30T05:44:00.075ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20002024-03-30T05:44:00.076ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
20012024-03-30T05:44:00.093ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20022024-03-30T05:44:00.093ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
20032024-03-30T05:44:00.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
20042024-03-30T05:44:00.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
20052024-03-30T05:44:00.128ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20062024-03-30T05:44:00.128ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
20072024-03-30T05:44:00.145ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20082024-03-30T05:44:00.145ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
20092024-03-30T05:44:00.163ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20102024-03-30T05:44:00.163ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
20112024-03-30T05:44:00.181ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20122024-03-30T05:44:00.181ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
20132024-03-30T05:44:00.198ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20142024-03-30T05:44:00.198ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
20152024-03-30T05:44:00.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20162024-03-30T05:44:00.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
20172024-03-30T05:44:00.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20182024-03-30T05:44:00.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
20192024-03-30T05:44:00.254ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20202024-03-30T05:44:00.254ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
20212024-03-30T05:44:00.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20222024-03-30T05:44:00.271ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
20232024-03-30T05:44:00.290ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20242024-03-30T05:44:00.290ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
20252024-03-30T05:44:00.290ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
20262024-03-30T05:44:00.290ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20272024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
20282024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
20292024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
20302024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
20312024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
20322024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
20332024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
20342024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
20352024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
20362024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
20372024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
20382024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
20392024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
20402024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
20412024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
20422024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
20432024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
20442024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
20452024-03-30T05:44:00.384ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
20462024-03-30T05:44:00.384ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 457910 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 5ab38bb7-691d-4ad0-bedb-21fb7ceb1c58 response_code = 500 uri = /omicron-zones
20472024-03-30T05:44:00.385ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5ab38bb7-691d-4ad0-bedb-21fb7ceb1c58", "content-length": "124", "date": "Sat, 30 Mar 2024 05:44:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5ab38bb7-691d-4ad0-bedb-21fb7ceb1c58" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 1.691299746s sled_address = [fd00:1122:3344:101::1]:12345
20482024-03-30T05:44:02.077ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
20492024-03-30T05:44:02.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20502024-03-30T05:44:02.077ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
20512024-03-30T05:44:02.099ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20522024-03-30T05:44:02.099ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
20532024-03-30T05:44:02.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20542024-03-30T05:44:02.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
20552024-03-30T05:44:02.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20562024-03-30T05:44:02.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
20572024-03-30T05:44:02.156ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20582024-03-30T05:44:02.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
20592024-03-30T05:44:02.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20602024-03-30T05:44:02.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
20612024-03-30T05:44:02.191ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20622024-03-30T05:44:02.191ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
20632024-03-30T05:44:02.209ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20642024-03-30T05:44:02.209ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
20652024-03-30T05:44:02.227ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20662024-03-30T05:44:02.227ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
20672024-03-30T05:44:02.245ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20682024-03-30T05:44:02.245ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
20692024-03-30T05:44:02.264ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
20702024-03-30T05:44:02.264ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
20712024-03-30T05:44:02.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20722024-03-30T05:44:02.284ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
20732024-03-30T05:44:02.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20742024-03-30T05:44:02.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
20752024-03-30T05:44:02.320ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20762024-03-30T05:44:02.321ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
20772024-03-30T05:44:02.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20782024-03-30T05:44:02.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
20792024-03-30T05:44:02.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20802024-03-30T05:44:02.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
20812024-03-30T05:44:02.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20822024-03-30T05:44:02.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
20832024-03-30T05:44:02.393ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20842024-03-30T05:44:02.393ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
20852024-03-30T05:44:02.412ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20862024-03-30T05:44:02.412ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
20872024-03-30T05:44:02.430ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20882024-03-30T05:44:02.430ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
20892024-03-30T05:44:02.448ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20902024-03-30T05:44:02.448ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
20912024-03-30T05:44:02.448ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
20922024-03-30T05:44:02.448ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20932024-03-30T05:44:02.541ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
20942024-03-30T05:44:02.541ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
20952024-03-30T05:44:02.541ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
20962024-03-30T05:44:02.541ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
20972024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
20982024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
20992024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
21002024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
21012024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
21022024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
21032024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
21042024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
21052024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
21062024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
21072024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
21082024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
21092024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
21102024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
21112024-03-30T05:44:02.542ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
21122024-03-30T05:44:02.542ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 464743 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = b23a63be-d6c6-4537-a070-1ad126ee5856 response_code = 500 uri = /omicron-zones
21132024-03-30T05:44:02.542ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b23a63be-d6c6-4537-a070-1ad126ee5856", "content-length": "124", "date": "Sat, 30 Mar 2024 05:44:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b23a63be-d6c6-4537-a070-1ad126ee5856" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 3.46100635s sled_address = [fd00:1122:3344:101::1]:12345
21142024-03-30T05:44:06.005ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
21152024-03-30T05:44:06.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21162024-03-30T05:44:06.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
21172024-03-30T05:44:06.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21182024-03-30T05:44:06.029ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
21192024-03-30T05:44:06.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21202024-03-30T05:44:06.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
21212024-03-30T05:44:06.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21222024-03-30T05:44:06.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
21232024-03-30T05:44:06.088ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21242024-03-30T05:44:06.088ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
21252024-03-30T05:44:06.108ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21262024-03-30T05:44:06.108ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
21272024-03-30T05:44:06.127ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21282024-03-30T05:44:06.127ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
21292024-03-30T05:44:06.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21302024-03-30T05:44:06.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
21312024-03-30T05:44:06.165ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21322024-03-30T05:44:06.165ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
21332024-03-30T05:44:06.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21342024-03-30T05:44:06.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
21352024-03-30T05:44:06.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
21362024-03-30T05:44:06.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
21372024-03-30T05:44:06.222ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21382024-03-30T05:44:06.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
21392024-03-30T05:44:06.240ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21402024-03-30T05:44:06.240ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
21412024-03-30T05:44:06.259ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21422024-03-30T05:44:06.259ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
21432024-03-30T05:44:06.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21442024-03-30T05:44:06.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
21452024-03-30T05:44:06.295ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21462024-03-30T05:44:06.295ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
21472024-03-30T05:44:06.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21482024-03-30T05:44:06.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
21492024-03-30T05:44:06.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21502024-03-30T05:44:06.330ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
21512024-03-30T05:44:06.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21522024-03-30T05:44:06.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
21532024-03-30T05:44:06.365ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21542024-03-30T05:44:06.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
21552024-03-30T05:44:06.382ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21562024-03-30T05:44:06.382ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
21572024-03-30T05:44:06.382ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
21582024-03-30T05:44:06.382ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21592024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
21602024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
21612024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
21622024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
21632024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
21642024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
21652024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
21662024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
21672024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
21682024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
21692024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
21702024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
21712024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
21722024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
21732024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
21742024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
21752024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
21762024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
21772024-03-30T05:44:06.473ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
21782024-03-30T05:44:06.473ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 468141 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = f8539a9a-ae0a-43dd-8a7c-0ed2ac3dc282 response_code = 500 uri = /omicron-zones
21792024-03-30T05:44:06.474ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f8539a9a-ae0a-43dd-8a7c-0ed2ac3dc282", "content-length": "124", "date": "Sat, 30 Mar 2024 05:44:06 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f8539a9a-ae0a-43dd-8a7c-0ed2ac3dc282" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 1.843988878s sled_address = [fd00:1122:3344:101::1]:12345
21802024-03-30T05:44:08.315ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
21812024-03-30T05:44:08.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21822024-03-30T05:44:08.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
21832024-03-30T05:44:08.337ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21842024-03-30T05:44:08.337ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
21852024-03-30T05:44:08.356ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21862024-03-30T05:44:08.356ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
21872024-03-30T05:44:08.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21882024-03-30T05:44:08.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
21892024-03-30T05:44:08.393ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21902024-03-30T05:44:08.393ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
21912024-03-30T05:44:08.411ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21922024-03-30T05:44:08.411ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
21932024-03-30T05:44:08.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21942024-03-30T05:44:08.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
21952024-03-30T05:44:08.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21962024-03-30T05:44:08.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
21972024-03-30T05:44:08.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21982024-03-30T05:44:08.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
21992024-03-30T05:44:08.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22002024-03-30T05:44:08.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
22012024-03-30T05:44:08.500ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
22022024-03-30T05:44:08.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
22032024-03-30T05:44:08.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22042024-03-30T05:44:08.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
22052024-03-30T05:44:08.538ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22062024-03-30T05:44:08.538ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
22072024-03-30T05:44:08.556ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22082024-03-30T05:44:08.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
22092024-03-30T05:44:08.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22102024-03-30T05:44:08.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
22112024-03-30T05:44:08.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22122024-03-30T05:44:08.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
22132024-03-30T05:44:08.610ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22142024-03-30T05:44:08.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
22152024-03-30T05:44:08.629ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22162024-03-30T05:44:08.631ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
22172024-03-30T05:44:08.647ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22182024-03-30T05:44:08.647ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
22192024-03-30T05:44:08.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22202024-03-30T05:44:08.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
22212024-03-30T05:44:08.681ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22222024-03-30T05:44:08.681ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
22232024-03-30T05:44:08.681ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
22242024-03-30T05:44:08.682ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22252024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
22262024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
22272024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
22282024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
22292024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
22302024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
22312024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
22322024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
22332024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
22342024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
22352024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
22362024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
22372024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
22382024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
22392024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
22402024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
22412024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
22422024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
22432024-03-30T05:44:08.769ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
22442024-03-30T05:44:08.769ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 455217 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 3aa678d7-35dc-48b9-b625-fb7df19ede3c response_code = 500 uri = /omicron-zones
22452024-03-30T05:44:08.769ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3aa678d7-35dc-48b9-b625-fb7df19ede3c", "content-length": "124", "date": "Sat, 30 Mar 2024 05:44:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3aa678d7-35dc-48b9-b625-fb7df19ede3c" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 5.408723479s sled_address = [fd00:1122:3344:101::1]:12345
22462024-03-30T05:44:12.605ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37758
22472024-03-30T05:44:12.606ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 49 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37758 req_id = 86d9c9d9-585b-4cf5-8475-3bc3ddb9696f response_code = 200 uri = /baseboard
22482024-03-30T05:44:14.165ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
22492024-03-30T05:44:14.167ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22502024-03-30T05:44:14.167ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
22512024-03-30T05:44:14.187ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22522024-03-30T05:44:14.187ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
22532024-03-30T05:44:14.207ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22542024-03-30T05:44:14.209ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
22552024-03-30T05:44:14.224ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22562024-03-30T05:44:14.224ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
22572024-03-30T05:44:14.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22582024-03-30T05:44:14.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
22592024-03-30T05:44:14.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22602024-03-30T05:44:14.261ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
22612024-03-30T05:44:14.279ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22622024-03-30T05:44:14.279ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
22632024-03-30T05:44:14.296ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22642024-03-30T05:44:14.296ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
22652024-03-30T05:44:14.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22662024-03-30T05:44:14.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
22672024-03-30T05:44:14.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22682024-03-30T05:44:14.333ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
22692024-03-30T05:44:14.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
22702024-03-30T05:44:14.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
22712024-03-30T05:44:14.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22722024-03-30T05:44:14.368ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
22732024-03-30T05:44:14.384ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22742024-03-30T05:44:14.384ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
22752024-03-30T05:44:14.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22762024-03-30T05:44:14.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
22772024-03-30T05:44:14.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22782024-03-30T05:44:14.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
22792024-03-30T05:44:14.437ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22802024-03-30T05:44:14.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
22812024-03-30T05:44:14.455ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22822024-03-30T05:44:14.455ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
22832024-03-30T05:44:14.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22842024-03-30T05:44:14.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
22852024-03-30T05:44:14.490ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22862024-03-30T05:44:14.490ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
22872024-03-30T05:44:14.507ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22882024-03-30T05:44:14.507ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
22892024-03-30T05:44:14.525ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22902024-03-30T05:44:14.525ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
22912024-03-30T05:44:14.526ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
22922024-03-30T05:44:14.526ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22932024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
22942024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
22952024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
22962024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
22972024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
22982024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
22992024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
23002024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
23012024-03-30T05:44:14.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
23022024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
23032024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
23042024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
23052024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
23062024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
23072024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
23082024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
23092024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
23102024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
23112024-03-30T05:44:14.615ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
23122024-03-30T05:44:14.615ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized), ("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 450206 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = bc68910c-55e4-4eff-a3bb-dfdb8a3a2fd5 response_code = 500 uri = /omicron-zones
23132024-03-30T05:44:14.615ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bc68910c-55e4-4eff-a3bb-dfdb8a3a2fd5", "content-length": "124", "date": "Sat, 30 Mar 2024 05:44:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bc68910c-55e4-4eff-a3bb-dfdb8a3a2fd5" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 4.034242589s sled_address = [fd00:1122:3344:101::1]:12345
23142024-03-30T05:44:18.639ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
23152024-03-30T05:44:18.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23162024-03-30T05:44:18.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
23172024-03-30T05:44:18.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23182024-03-30T05:44:18.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
23192024-03-30T05:44:18.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23202024-03-30T05:44:18.680ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
23212024-03-30T05:44:18.699ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23222024-03-30T05:44:18.699ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
23232024-03-30T05:44:18.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23242024-03-30T05:44:18.718ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
23252024-03-30T05:44:18.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23262024-03-30T05:44:18.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
23272024-03-30T05:44:18.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23282024-03-30T05:44:18.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
23292024-03-30T05:44:18.772ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23302024-03-30T05:44:18.772ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
23312024-03-30T05:44:18.790ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23322024-03-30T05:44:18.790ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
23332024-03-30T05:44:18.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23342024-03-30T05:44:18.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
23352024-03-30T05:44:18.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
23362024-03-30T05:44:18.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
23372024-03-30T05:44:18.844ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23382024-03-30T05:44:18.844ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
23392024-03-30T05:44:18.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23402024-03-30T05:44:18.863ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
23412024-03-30T05:44:18.881ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23422024-03-30T05:44:18.881ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
23432024-03-30T05:44:18.900ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23442024-03-30T05:44:18.900ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
23452024-03-30T05:44:18.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23462024-03-30T05:44:18.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
23472024-03-30T05:44:18.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23482024-03-30T05:44:18.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
23492024-03-30T05:44:18.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23502024-03-30T05:44:18.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
23512024-03-30T05:44:18.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23522024-03-30T05:44:18.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
23532024-03-30T05:44:18.993ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23542024-03-30T05:44:18.993ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
23552024-03-30T05:44:19.012ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
23562024-03-30T05:44:19.012ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
23572024-03-30T05:44:19.012ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
23582024-03-30T05:44:19.012ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
23592024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
23602024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
23612024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
23622024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
23632024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
23642024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
23652024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
23662024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
23672024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
23682024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
23692024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
23702024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
23712024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
23722024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
23732024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
23742024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
23752024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
23762024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
23772024-03-30T05:44:19.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3001 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
23782024-03-30T05:44:19.105ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc", TimeNotSynchronized), ("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5", TimeNotSynchronized), ("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8", TimeNotSynchronized), ("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd", TimeNotSynchronized), ("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523", TimeNotSynchronized), ("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc", TimeNotSynchronized), ("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f", TimeNotSynchronized), ("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6", TimeNotSynchronized), ("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4", TimeNotSynchronized), ("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82", TimeNotSynchronized), ("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859", TimeNotSynchronized), ("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d", TimeNotSynchronized), ("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88", TimeNotSynchronized), ("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce", TimeNotSynchronized), ("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72", TimeNotSynchronized), ("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76", TimeNotSynchronized), ("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528", TimeNotSynchronized), ("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9", TimeNotSynchronized), ("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:837 latency_us = 467934 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = db34b5ee-7b53-44b7-8ac0-f07eda92f9b4 response_code = 500 uri = /omicron-zones
23792024-03-30T05:44:19.106ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "db34b5ee-7b53-44b7-8ac0-f07eda92f9b4", "content-length": "124", "date": "Sat, 30 Mar 2024 05:44:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "db34b5ee-7b53-44b7-8ac0-f07eda92f9b4" } file = sled-agent/src/rack_setup/service.rs:465 retry_after = 6.79230886s sled_address = [fd00:1122:3344:101::1]:12345
23802024-03-30T05:44:25.881ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: c037c9e2-43a2-432a-bcd7-352e08f26490, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 655db966-87f1-48aa-8868-000f036af1b2, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: a01b5bfc-3390-497f-96d2-052158f09a46, kind: Service { id: 2168e6f2-fbb0-4d6d-8811-9fd75847a2a7 }, name: Name("ntp-2168e6f2-fbb0-4d6d-8811-9fd75847a2a7"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 143, 167])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.3.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: f7311d36-617f-4f72-be7a-3427dc69c306, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f597bf55-7c6a-456c-8e41-286fd5037349, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 57425cdc-844b-42e3-b45f-c8fb7871161f, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b6a17b0f-0906-4eb9-94c3-528d000f68dc, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: f4d5df0a-748d-40ab-8235-7c927aae50ce, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 23c9f9de-3736-4878-ba3b-118cebaacb82, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: a684a100-6031-4b97-8de3-41e15037fbd5, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 648e5f66-03f8-489e-a2bb-91cee8654a72, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External } } } }, OmicronZoneConfig { id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External } } } }, OmicronZoneConfig { id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External } } } }, OmicronZoneConfig { id: 80d3c328-30b9-4b19-852b-d0002f908859, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External } } } }, OmicronZoneConfig { id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External } } } }, OmicronZoneConfig { id: 06241231-d15c-4adb-ae5b-e8e364e35528, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External } } } }, OmicronZoneConfig { id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External } } } }, OmicronZoneConfig { id: 57f3dc22-0a8a-40ec-a7b7-137531501134, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External } } } }, OmicronZoneConfig { id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:427 sled_address = [fd00:1122:3344:101::1]:12345
23812024-03-30T05:44:25.882ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23822024-03-30T05:44:25.882ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 55e16fed-0b39-4c9e-94aa-dbf1dfed9795, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: InternalDns }, responder: ... }
23832024-03-30T05:44:25.906ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23842024-03-30T05:44:25.906ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c037c9e2-43a2-432a-bcd7-352e08f26490, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: InternalDns }, responder: ... }
23852024-03-30T05:44:25.926ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23862024-03-30T05:44:25.926ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 655db966-87f1-48aa-8868-000f036af1b2, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: InternalDns }, responder: ... }
23872024-03-30T05:44:25.945ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23882024-03-30T05:44:25.945ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7311d36-617f-4f72-be7a-3427dc69c306, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: CockroachDb }, responder: ... }
23892024-03-30T05:44:25.965ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23902024-03-30T05:44:25.965ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f597bf55-7c6a-456c-8e41-286fd5037349, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: CockroachDb }, responder: ... }
23912024-03-30T05:44:25.983ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23922024-03-30T05:44:25.983ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ddf7c878-622c-4bc8-8b0b-5a158ff3b198, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: CockroachDb }, responder: ... }
23932024-03-30T05:44:26.001ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23942024-03-30T05:44:26.001ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9d0139eb-7e07-43fa-92d7-f8a6676bf465, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: CockroachDb }, responder: ... }
23952024-03-30T05:44:26.019ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23962024-03-30T05:44:26.019ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: f7108ea5-965b-4e70-a9a0-7ffd5d13885f, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: CockroachDb }, responder: ... }
23972024-03-30T05:44:26.036ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23982024-03-30T05:44:26.036ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57425cdc-844b-42e3-b45f-c8fb7871161f, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: ExternalDns }, responder: ... }
23992024-03-30T05:44:26.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24002024-03-30T05:44:26.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: ExternalDns }, responder: ... }
24012024-03-30T05:44:26.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
24022024-03-30T05:44:26.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8bf3945e-e049-4e64-8e87-3d72b2744d88, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Clickhouse }, responder: ... }
24032024-03-30T05:44:26.092ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24042024-03-30T05:44:26.092ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 648e5f66-03f8-489e-a2bb-91cee8654a72, dataset_name: DatasetName { pool_name: ZpoolName { id: 82347231-68ab-4934-8007-b28d9a3a3777, kind: External }, kind: Crucible }, responder: ... }
24052024-03-30T05:44:26.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24062024-03-30T05:44:26.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 525c4f1a-5d69-4ecf-a43b-162fb1134523, dataset_name: DatasetName { pool_name: ZpoolName { id: 28899eb6-914e-44b6-b77e-9d82d92186d6, kind: External }, kind: Crucible }, responder: ... }
24072024-03-30T05:44:26.128ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24082024-03-30T05:44:26.128ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0a93b238-dc4f-4234-98ad-cf545c1b8ea6, dataset_name: DatasetName { pool_name: ZpoolName { id: 6de8542c-123a-4abc-9a88-e72f1063a9f4, kind: External }, kind: Crucible }, responder: ... }
24092024-03-30T05:44:26.147ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24102024-03-30T05:44:26.147ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 80d3c328-30b9-4b19-852b-d0002f908859, dataset_name: DatasetName { pool_name: ZpoolName { id: e43353e3-7e66-4e6f-bd56-19f8143ee8b5, kind: External }, kind: Crucible }, responder: ... }
24112024-03-30T05:44:26.165ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24122024-03-30T05:44:26.165ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47a9dbe8-5005-4fee-bb98-58944f67aa76, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c3019ca-328c-47ab-9982-06e092199db8, kind: External }, kind: Crucible }, responder: ... }
24132024-03-30T05:44:26.183ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24142024-03-30T05:44:26.183ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 06241231-d15c-4adb-ae5b-e8e364e35528, dataset_name: DatasetName { pool_name: ZpoolName { id: 3c92b6bd-8180-45a3-9123-a99ddc649159, kind: External }, kind: Crucible }, responder: ... }
24152024-03-30T05:44:26.204ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24162024-03-30T05:44:26.204ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 22aed52b-11e9-4a7d-bebf-3356da024fc9, dataset_name: DatasetName { pool_name: ZpoolName { id: ea19d3d4-e058-46fb-981b-9ccb255f563c, kind: External }, kind: Crucible }, responder: ... }
24172024-03-30T05:44:26.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24182024-03-30T05:44:26.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 57f3dc22-0a8a-40ec-a7b7-137531501134, dataset_name: DatasetName { pool_name: ZpoolName { id: e22df0ad-bbf8-4902-93d6-e67ff23c3234, kind: External }, kind: Crucible }, responder: ... }
24192024-03-30T05:44:26.238ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24202024-03-30T05:44:26.238ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6eebe2cc-6b03-4cdd-bd4d-26af275d74bc, dataset_name: DatasetName { pool_name: ZpoolName { id: c7e0e489-6c2b-472e-8a1f-efe83bb548dd, kind: External }, kind: Crucible }, responder: ... }
24212024-03-30T05:44:26.255ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24222024-03-30T05:44:26.255ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
24232024-03-30T05:44:26.256ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
24242024-03-30T05:44:26.256ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24252024-03-30T05:44:26.343ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
24262024-03-30T05:44:26.379ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
24272024-03-30T05:44:26.380ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24282024-03-30T05:44:26.380ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24292024-03-30T05:44:26.404ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
24302024-03-30T05:44:26.404ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24312024-03-30T05:44:26.404ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24322024-03-30T05:44:26.429ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24332024-03-30T05:44:26.429ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24342024-03-30T05:44:26.429ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24352024-03-30T05:44:26.579ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
24362024-03-30T05:44:26.580ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24372024-03-30T05:44:26.580ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24382024-03-30T05:44:26.594ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:258
24392024-03-30T05:44:26.595ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24402024-03-30T05:44:26.595ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24412024-03-30T05:44:26.630ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24422024-03-30T05:44:26.630ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24432024-03-30T05:44:26.630ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24442024-03-30T05:44:26.642ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134 file = illumos-utils/src/zone.rs:315
24452024-03-30T05:44:26.647ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc file = illumos-utils/src/zone.rs:315
24462024-03-30T05:44:26.652ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88 file = illumos-utils/src/zone.rs:315
24472024-03-30T05:44:26.657ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82 file = illumos-utils/src/zone.rs:315
24482024-03-30T05:44:26.663ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5 file = illumos-utils/src/zone.rs:315
24492024-03-30T05:44:26.668ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce file = illumos-utils/src/zone.rs:315
24502024-03-30T05:44:26.673ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76 file = illumos-utils/src/zone.rs:315
24512024-03-30T05:44:26.808ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134 file = illumos-utils/src/zone.rs:354
24522024-03-30T05:44:26.813ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc file = illumos-utils/src/zone.rs:354
24532024-03-30T05:44:26.818ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88 file = illumos-utils/src/zone.rs:354
24542024-03-30T05:44:26.823ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82 file = illumos-utils/src/zone.rs:354
24552024-03-30T05:44:26.827ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5 file = illumos-utils/src/zone.rs:354
24562024-03-30T05:44:26.832ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce file = illumos-utils/src/zone.rs:354
24572024-03-30T05:44:26.837ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76 file = illumos-utils/src/zone.rs:354
24582024-03-30T05:44:26.842ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9 file = illumos-utils/src/zone.rs:315
24592024-03-30T05:44:26.846ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6 file = illumos-utils/src/zone.rs:315
24602024-03-30T05:44:26.851ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc file = illumos-utils/src/zone.rs:315
24612024-03-30T05:44:26.856ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72 file = illumos-utils/src/zone.rs:315
24622024-03-30T05:44:26.875ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523 file = illumos-utils/src/zone.rs:315
24632024-03-30T05:44:26.880ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859 file = illumos-utils/src/zone.rs:315
24642024-03-30T05:44:26.885ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528 file = illumos-utils/src/zone.rs:315
24652024-03-30T05:44:26.891ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9 file = illumos-utils/src/zone.rs:354
24662024-03-30T05:44:26.896ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6 file = illumos-utils/src/zone.rs:354
24672024-03-30T05:44:26.902ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc file = illumos-utils/src/zone.rs:354
24682024-03-30T05:44:26.906ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72 file = illumos-utils/src/zone.rs:354
24692024-03-30T05:44:26.911ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523 file = illumos-utils/src/zone.rs:354
24702024-03-30T05:44:26.916ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859 file = illumos-utils/src/zone.rs:354
24712024-03-30T05:44:26.920ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528 file = illumos-utils/src/zone.rs:354
24722024-03-30T05:44:26.976ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
24732024-03-30T05:44:27.015ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
24742024-03-30T05:44:27.058ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
24752024-03-30T05:44:27.100ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
24762024-03-30T05:44:27.133ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:275
24772024-03-30T05:44:27.134ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
24782024-03-30T05:44:27.134ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
24792024-03-30T05:44:27.193ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 133, 194]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte1" } }
24802024-03-30T05:44:27.193ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1277 zone_type = nexus
24812024-03-30T05:44:27.196ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
24822024-03-30T05:44:27.196ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
24832024-03-30T05:44:27.252ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte2", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 254, 213]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte2" } }
24842024-03-30T05:44:27.252ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1277 zone_type = external_dns
24852024-03-30T05:44:27.254ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
24862024-03-30T05:44:27.254ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
24872024-03-30T05:44:27.309ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 195, 126]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte3" } }
24882024-03-30T05:44:27.310ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1277 zone_type = external_dns
24892024-03-30T05:44:27.312ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
24902024-03-30T05:44:27.312ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
24912024-03-30T05:44:27.364ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 246, 195]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte4" } }
24922024-03-30T05:44:27.364ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1277 zone_type = nexus
24932024-03-30T05:44:27.367ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:300
24942024-03-30T05:44:27.367ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:192
24952024-03-30T05:44:27.419ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 171, 116]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte5" } }
24962024-03-30T05:44:27.419ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1277 zone_type = nexus
24972024-03-30T05:44:27.420ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
24982024-03-30T05:44:27.420ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
24992024-03-30T05:44:27.428ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
25002024-03-30T05:44:27.431ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25012024-03-30T05:44:27.456ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
25022024-03-30T05:44:27.456ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
25032024-03-30T05:44:27.456ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
25042024-03-30T05:44:27.456ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
25052024-03-30T05:44:27.456ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
25062024-03-30T05:44:27.456ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
25072024-03-30T05:44:27.456ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
25082024-03-30T05:44:27.456ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
25092024-03-30T05:44:27.457ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
25102024-03-30T05:44:27.459ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
25112024-03-30T05:44:27.461ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
25122024-03-30T05:44:27.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25132024-03-30T05:44:27.461ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25142024-03-30T05:44:27.464ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.1.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
25152024-03-30T05:44:27.466ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25162024-03-30T05:44:27.466ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25172024-03-30T05:44:27.555ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d file = illumos-utils/src/zone.rs:315
25182024-03-30T05:44:27.560ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f file = illumos-utils/src/zone.rs:315
25192024-03-30T05:44:27.565ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 file = illumos-utils/src/zone.rs:315
25202024-03-30T05:44:27.571ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 file = illumos-utils/src/zone.rs:315
25212024-03-30T05:44:27.577ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd file = illumos-utils/src/zone.rs:315
25222024-03-30T05:44:27.583ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d file = illumos-utils/src/zone.rs:354
25232024-03-30T05:44:27.589ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f file = illumos-utils/src/zone.rs:354
25242024-03-30T05:44:27.595ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 file = illumos-utils/src/zone.rs:354
25252024-03-30T05:44:27.600ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 file = illumos-utils/src/zone.rs:354
25262024-03-30T05:44:27.607ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd file = illumos-utils/src/zone.rs:354
25272024-03-30T05:44:30.747ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::11"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25282024-03-30T05:44:30.748ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::10"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25292024-03-30T05:44:30.748ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
25302024-03-30T05:44:30.756ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::f"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25312024-03-30T05:44:30.757ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
25322024-03-30T05:44:30.786ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
25332024-03-30T05:44:30.915ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::18"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_ea19d3d4-e058-46fb-981b-9ccb255f563c/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="093e299e-c70c-49a4-b8d2-cdbc7519b070"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25342024-03-30T05:44:30.915ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::17"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_3c92b6bd-8180-45a3-9123-a99ddc649159/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="444b65ae-5031-4e1b-9878-1efb6b08ce20"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25352024-03-30T05:44:30.915ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
25362024-03-30T05:44:30.923ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::14"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_6de8542c-123a-4abc-9a88-e72f1063a9f4/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="d3ad92e9-addf-47ad-ae73-22b9743a83f5"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25372024-03-30T05:44:30.923ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::12"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_82347231-68ab-4934-8007-b28d9a3a3777/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="8cd13b47-e572-4625-8022-57c0f24dfaa0"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25382024-03-30T05:44:30.924ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::13"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_28899eb6-914e-44b6-b77e-9d82d92186d6/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="a9995da8-953f-4ad9-8cf8-b484e080974b"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25392024-03-30T05:44:30.924ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::15"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_e43353e3-7e66-4e6f-bd56-19f8143ee8b5/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="c75657d7-3cd4-435d-981a-11201eeec233"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25402024-03-30T05:44:30.924ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
25412024-03-30T05:44:30.931ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::19"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_e22df0ad-bbf8-4902-93d6-e67ff23c3234/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="96d123ae-b766-40aa-b115-103d9ba68b91"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25422024-03-30T05:44:30.931ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_c7e0e489-6c2b-472e-8a1f-efe83bb548dd/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="ee6966f6-a047-4b26-be03-a11646ea767c"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25432024-03-30T05:44:30.932ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::16"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_5c3019ca-328c-47ab-9982-06e092199db8/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="3f5dbb2e-6848-41fc-ba03-a3eb231fa74a"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25442024-03-30T05:44:30.932ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
25452024-03-30T05:44:30.939ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
25462024-03-30T05:44:30.945ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
25472024-03-30T05:44:30.952ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
25482024-03-30T05:44:30.959ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
25492024-03-30T05:44:30.965ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
25502024-03-30T05:44:30.973ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
25512024-03-30T05:44:31.326ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::d"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value="b6a17b0f-0906-4eb9-94c3-528d000f68dc"/> <propval type="astring" name="address" value="[fd00:1122:3344:101::d]:12223"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
25522024-03-30T05:44:31.327ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
25532024-03-30T05:44:31.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 48.854819ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
25542024-03-30T05:44:31.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 29.875273ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
25552024-03-30T05:44:31.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 40.580244ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
25562024-03-30T05:44:31.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 62.925229ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
25572024-03-30T05:44:31.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 67.798406ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
25582024-03-30T05:44:31.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 62.734311ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
25592024-03-30T05:44:32.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 72.366857ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
25602024-03-30T05:44:32.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 54.412418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
25612024-03-30T05:44:41.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 55.11272ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
25622024-03-30T05:44:41.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 41.500283ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
25632024-03-30T05:44:41.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 65.224797ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
25642024-03-30T05:44:41.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 70.296942ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
25652024-03-30T05:44:41.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 35.705639ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
25662024-03-30T05:44:42.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 126.727812ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
25672024-03-30T05:44:42.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 87.328966ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
25682024-03-30T05:44:42.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 144.882375ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
25692024-03-30T05:44:42.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 95.155341ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
25702024-03-30T05:44:42.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 72.610248ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
25712024-03-30T05:44:42.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 83.881003ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
25722024-03-30T05:44:42.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 115.95931ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
25732024-03-30T05:44:42.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 95.143748ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
25742024-03-30T05:44:42.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 124.505321ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
25752024-03-30T05:44:42.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 106.449786ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
25762024-03-30T05:44:42.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 105.231135ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
25772024-03-30T05:44:42.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 65.211443ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
25782024-03-30T05:44:42.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 50.75902ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
25792024-03-30T05:44:42.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 250.999964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
25802024-03-30T05:44:42.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 160.761524ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
25812024-03-30T05:44:42.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 242.321793ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
25822024-03-30T05:44:42.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 194.089351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
25832024-03-30T05:44:42.457ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 219.994138ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
25842024-03-30T05:44:42.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 212.469413ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
25852024-03-30T05:44:42.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 127.550512ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
25862024-03-30T05:44:42.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 268.821116ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
25872024-03-30T05:44:42.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 199.333164ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
25882024-03-30T05:44:42.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 253.552384ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
25892024-03-30T05:44:42.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 235.896515ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
25902024-03-30T05:44:42.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 192.323902ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
25912024-03-30T05:44:42.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 100.072973ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
25922024-03-30T05:44:42.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 454.688091ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
25932024-03-30T05:44:42.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 319.944679ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
25942024-03-30T05:44:42.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 372.917834ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
25952024-03-30T05:44:42.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 399.307918ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
25962024-03-30T05:44:42.701ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 288.464056ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
25972024-03-30T05:44:42.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 356.339504ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
25982024-03-30T05:44:42.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 485.043168ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
25992024-03-30T05:44:42.733ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56239
26002024-03-30T05:44:42.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 241.896835ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
26012024-03-30T05:44:42.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 364.191695ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
26022024-03-30T05:44:42.779ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 34239 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56239 req_id = 0f5c119f-9e68-43c5-b9b2-a68fde39eb43 response_code = 200 uri = /baseboard
26032024-03-30T05:44:42.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 401.911969ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
26042024-03-30T05:44:42.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 259.358962ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
26052024-03-30T05:44:42.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 370.154221ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26062024-03-30T05:44:42.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 290.21036ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
26072024-03-30T05:44:42.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 1.006306015s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
26082024-03-30T05:44:43.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 1.039892101s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
26092024-03-30T05:44:43.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 1.15198777s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
26102024-03-30T05:44:43.064ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 812.851166ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
26112024-03-30T05:44:43.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 1.087742599s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
26122024-03-30T05:44:43.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 769.566436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
26132024-03-30T05:44:43.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 853.434585ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
26142024-03-30T05:44:43.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 1.064721091s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
26152024-03-30T05:44:43.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 786.543892ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
26162024-03-30T05:44:43.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 1.120239206s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
26172024-03-30T05:44:43.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 502.344179ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26182024-03-30T05:44:43.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 626.318066ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
26192024-03-30T05:44:43.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 408.09342ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
26202024-03-30T05:44:43.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 1.184453253s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
26212024-03-30T05:44:43.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 692.218586ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26222024-03-30T05:44:43.811ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService25"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::8"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte2"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::8]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.5:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26232024-03-30T05:44:43.811ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService26"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::9"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte3"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::9]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.6:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26242024-03-30T05:44:43.814ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
26252024-03-30T05:44:43.821ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
26262024-03-30T05:44:44.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 860.964629ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
26272024-03-30T05:44:44.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 1.48850656s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
26282024-03-30T05:44:44.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 1.238240603s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
26292024-03-30T05:44:44.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 1.214371553s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
26302024-03-30T05:44:44.502ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 1.231055732s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
26312024-03-30T05:44:44.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 1.499581011s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
26322024-03-30T05:44:44.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 719.235483ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
26332024-03-30T05:44:44.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 956.387277ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
26342024-03-30T05:44:44.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 978.915256ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
26352024-03-30T05:44:44.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 1.139395781s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26362024-03-30T05:44:46.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 1.084220078s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
26372024-03-30T05:44:46.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 518.084503ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
26382024-03-30T05:44:46.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 930.126886ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
26392024-03-30T05:44:46.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 1.004925005s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
26402024-03-30T05:44:46.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 1.460067721s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
26412024-03-30T05:44:46.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 1.356575561s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
26422024-03-30T05:44:46.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 1.340260585s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
26432024-03-30T05:44:46.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 766.971167ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
26442024-03-30T05:44:46.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 511.922532ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
26452024-03-30T05:44:46.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 509.629317ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
26462024-03-30T05:44:46.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 1.446829192s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
26472024-03-30T05:44:46.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 722.75616ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
26482024-03-30T05:44:46.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 651.824507ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26492024-03-30T05:44:46.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 56.448242ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
26502024-03-30T05:44:46.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 34.800518ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
26512024-03-30T05:44:46.591ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 108.533546ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
26522024-03-30T05:44:46.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 102.964477ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
26532024-03-30T05:44:46.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 152.912413ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
26542024-03-30T05:44:46.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 217.783563ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
26552024-03-30T05:44:46.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 1.45192266s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
26562024-03-30T05:44:46.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 216.511509ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
26572024-03-30T05:44:46.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 1.388223128s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
26582024-03-30T05:44:47.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 386.307266ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
26592024-03-30T05:44:47.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 954.083643ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
26602024-03-30T05:44:47.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 544.180899ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
26612024-03-30T05:44:47.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 515.880689ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26622024-03-30T05:44:47.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 713.473874ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
26632024-03-30T05:44:47.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 928.793746ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
26642024-03-30T05:44:47.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 1.251418421s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
26652024-03-30T05:44:47.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 622.624021ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
26662024-03-30T05:44:47.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 1.115550903s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
26672024-03-30T05:44:47.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 498.286868ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
26682024-03-30T05:44:47.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 842.775261ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26692024-03-30T05:44:47.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 856.942977ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
26702024-03-30T05:44:47.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 898.314049ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
26712024-03-30T05:44:47.808ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 1.275246825s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
26722024-03-30T05:44:47.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 577.148565ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
26732024-03-30T05:44:47.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 960.391527ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
26742024-03-30T05:44:47.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 607.053831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
26752024-03-30T05:44:47.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 1.34827484s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
26762024-03-30T05:44:48.009ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 577.839981ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
26772024-03-30T05:44:48.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 1.193741162s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
26782024-03-30T05:44:48.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 1.481247608s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
26792024-03-30T05:44:48.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 501.942024ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
26802024-03-30T05:44:48.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 1.070280999s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
26812024-03-30T05:44:48.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 876.693067ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
26822024-03-30T05:44:48.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 852.957789ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
26832024-03-30T05:44:48.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 920.8774ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
26842024-03-30T05:44:48.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 500.703269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26852024-03-30T05:44:48.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 867.086497ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
26862024-03-30T05:44:48.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 1.126690302s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
26872024-03-30T05:44:48.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 791.073223ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
26882024-03-30T05:44:48.674ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService28"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::c"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte4"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26892024-03-30T05:44:48.675ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
26902024-03-30T05:44:48.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 742.43211ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
26912024-03-30T05:44:48.763ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService27"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte5"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26922024-03-30T05:44:48.763ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService24"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte1"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
26932024-03-30T05:44:48.763ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
26942024-03-30T05:44:48.772ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
26952024-03-30T05:44:48.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 648.956655ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
26962024-03-30T05:44:49.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 525.717326ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
26972024-03-30T05:44:49.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 1.109523416s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
26982024-03-30T05:44:49.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 1.133010599s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
26992024-03-30T05:44:49.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 926.191246ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
27002024-03-30T05:44:49.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 72.051078ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
27012024-03-30T05:44:49.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 1.009743559s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
27022024-03-30T05:44:49.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 1.195437473s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
27032024-03-30T05:44:49.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 31.059989ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
27042024-03-30T05:44:49.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 40.364055ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
27052024-03-30T05:44:49.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 929.285309ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
27062024-03-30T05:44:49.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 74.06991ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
27072024-03-30T05:44:49.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 87.433779ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
27082024-03-30T05:44:49.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 864.665969ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
27092024-03-30T05:44:49.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 86.519606ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
27102024-03-30T05:44:49.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 1.357652308s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
27112024-03-30T05:44:49.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 625.491586ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
27122024-03-30T05:44:49.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 1.088387289s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
27132024-03-30T05:44:49.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 810.693438ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
27142024-03-30T05:44:49.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 284.394451ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
27152024-03-30T05:44:49.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 758.48917ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
27162024-03-30T05:44:49.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 113.861558ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
27172024-03-30T05:44:49.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 735.545721ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
27182024-03-30T05:44:49.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 206.751449ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
27192024-03-30T05:44:49.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 786.794678ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
27202024-03-30T05:44:49.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 559.924127ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
27212024-03-30T05:44:49.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 933.752032ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
27222024-03-30T05:44:49.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 407.740277ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
27232024-03-30T05:44:49.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 240.675734ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
27242024-03-30T05:44:50.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 674.712419ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
27252024-03-30T05:44:50.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 1.193386916s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
27262024-03-30T05:44:50.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 1.342913954s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
27272024-03-30T05:44:50.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 621.319614ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
27282024-03-30T05:44:50.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 522.068889ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
27292024-03-30T05:44:50.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 826.340998ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
27302024-03-30T05:44:50.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 1.110576301s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
27312024-03-30T05:44:50.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 590.321521ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
27322024-03-30T05:44:50.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 892.744747ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
27332024-03-30T05:44:50.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 941.866176ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
27342024-03-30T05:44:50.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 958.934006ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
27352024-03-30T05:44:50.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 1.395068113s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
27362024-03-30T05:44:50.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 1.122878226s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
27372024-03-30T05:44:50.411ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::e"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
27382024-03-30T05:44:50.412ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
27392024-03-30T05:44:50.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 873.205795ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
27402024-03-30T05:44:50.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 1.383492687s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
27412024-03-30T05:44:50.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 1.318778892s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
27422024-03-30T05:44:50.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 49.756494ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
27432024-03-30T05:44:50.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 663.52308ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
27442024-03-30T05:44:50.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 67.084419ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
27452024-03-30T05:44:50.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 1.350247651s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
27462024-03-30T05:44:50.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 552.743464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
27472024-03-30T05:44:50.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 281.738688ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
27482024-03-30T05:44:50.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 598.703891ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
27492024-03-30T05:44:50.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 1.054460542s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
27502024-03-30T05:44:51.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 1.001199931s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
27512024-03-30T05:44:51.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 647.878086ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
27522024-03-30T05:44:51.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 561.959446ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
27532024-03-30T05:44:51.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 1.259055041s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
27542024-03-30T05:44:51.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 757.126852ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
27552024-03-30T05:44:51.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 1.14937169s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
27562024-03-30T05:44:51.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 1.225128082s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
27572024-03-30T05:44:51.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 1.069677643s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
27582024-03-30T05:44:51.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 1.244455621s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
27592024-03-30T05:44:51.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 928.461155ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
27602024-03-30T05:44:51.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 1.212850813s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
27612024-03-30T05:44:51.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 846.90682ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
27622024-03-30T05:44:51.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 1.265656003s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
27632024-03-30T05:44:51.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 531.887776ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
27642024-03-30T05:44:51.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 465.163202ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
27652024-03-30T05:44:51.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 1.368238215s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
27662024-03-30T05:44:51.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 957.667099ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
27672024-03-30T05:44:51.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 753.571852ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
27682024-03-30T05:44:51.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 1.4630397s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
27692024-03-30T05:44:52.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 1.495544305s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
27702024-03-30T05:44:52.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 879.913956ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
27712024-03-30T05:44:52.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 513.943085ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
27722024-03-30T05:44:52.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 937.768ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
27732024-03-30T05:44:52.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 890.475846ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
27742024-03-30T05:44:52.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 1.148184701s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
27752024-03-30T05:44:52.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 1.330642865s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
27762024-03-30T05:44:52.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 1.2952067s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
27772024-03-30T05:44:52.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 1.111690096s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
27782024-03-30T05:44:52.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 673.489828ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
27792024-03-30T05:44:52.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 779.244837ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
27802024-03-30T05:44:52.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 1.261988166s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
27812024-03-30T05:44:52.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 1.15123934s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
27822024-03-30T05:44:52.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 600.841797ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
27832024-03-30T05:44:52.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 1.469196731s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
27842024-03-30T05:44:52.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 1.488220743s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
27852024-03-30T05:44:52.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 914.508387ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
27862024-03-30T05:44:52.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 1.188288806s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
27872024-03-30T05:44:52.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 1.018185023s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
27882024-03-30T05:44:53.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 1.491739243s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
27892024-03-30T05:44:53.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 631.053681ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
27902024-03-30T05:44:53.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 975.001072ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
27912024-03-30T05:44:53.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76") failed: Property not found. retry in 1.398144517s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
27922024-03-30T05:44:53.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5") failed: Property not found. retry in 975.028002ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
27932024-03-30T05:44:53.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 1.439119584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
27942024-03-30T05:44:53.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 746.398795ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
27952024-03-30T05:44:53.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6") failed: Property not found. retry in 1.164890165s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
27962024-03-30T05:44:53.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 1.101945469s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
27972024-03-30T05:44:53.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc") failed: Property not found. retry in 1.018729259s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
27982024-03-30T05:44:53.701ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523") failed: Property not found. retry in 1.452519585s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
27992024-03-30T05:44:53.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134") failed: Property not found. retry in 1.407177838s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
28002024-03-30T05:44:53.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc") failed: Property not found. retry in 1.429982148s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
28012024-03-30T05:44:53.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82") failed: Property not found. retry in 928.333826ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
28022024-03-30T05:44:53.788ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 586.17688ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
28032024-03-30T05:44:53.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce") failed: Property not found. retry in 1.015024224s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
28042024-03-30T05:44:54.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 709.687754ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
28052024-03-30T05:44:54.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9") failed: Property not found. retry in 1.21256322s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
28062024-03-30T05:44:54.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72") failed: Property not found. retry in 979.147789ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
28072024-03-30T05:44:54.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 858.598696ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
28082024-03-30T05:44:54.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 629.816847ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
28092024-03-30T05:44:54.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528") failed: Property not found. retry in 608.188628ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
28102024-03-30T05:44:54.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 1.255936753s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
28112024-03-30T05:44:54.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 642.168051ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
28122024-03-30T05:44:54.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 831.394635ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
28132024-03-30T05:44:54.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 1.304996291s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
28142024-03-30T05:44:54.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859") failed: Property not found. retry in 1.276663338s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
28152024-03-30T05:44:54.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 929.042317ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
28162024-03-30T05:44:55.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 685.095962ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
28172024-03-30T05:44:55.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 1.43124304s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
28182024-03-30T05:44:55.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 980.084573ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
28192024-03-30T05:44:55.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 901.831747ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
28202024-03-30T05:44:55.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 1.041177085s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
28212024-03-30T05:44:55.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 1.346344462s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
28222024-03-30T05:44:56.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 658.92591ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
28232024-03-30T05:44:56.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f") failed: Property not found. retry in 1.411274044s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
28242024-03-30T05:44:56.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8") failed: Property not found. retry in 1.043427088s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
28252024-03-30T05:44:56.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 1.148763911s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
28262024-03-30T05:44:56.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 641.737632ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
28272024-03-30T05:44:56.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 1.273473164s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
28282024-03-30T05:44:57.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 1.011669798s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
28292024-03-30T05:44:57.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 637.038104ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
28302024-03-30T05:44:57.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4") failed: Property not found. retry in 843.893913ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
28312024-03-30T05:44:58.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d") failed: Property not found. retry in 912.540349ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
28322024-03-30T05:44:58.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd") failed: Property not found. retry in 1.191569795s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
28332024-03-30T05:44:58.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 693.127848ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
28342024-03-30T05:44:58.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88") failed: Property not found. retry in 792.693202ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
28352024-03-30T05:44:59.318ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
28362024-03-30T05:44:59.721ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_nexus_3b85f16b-9ecd-4867-ae56-7f8a1f492a4d
28372024-03-30T05:44:59.721ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_57f3dc22-0a8a-40ec-a7b7-137531501134
28382024-03-30T05:44:59.721ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_6eebe2cc-6b03-4cdd-bd4d-26af275d74bc
28392024-03-30T05:44:59.721ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_external_dns_57425cdc-844b-42e3-b45f-c8fb7871161f
28402024-03-30T05:44:59.721ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_clickhouse_8bf3945e-e049-4e64-8e87-3d72b2744d88
28412024-03-30T05:44:59.721ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_pantry_23c9f9de-3736-4878-ba3b-118cebaacb82
28422024-03-30T05:44:59.721ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_external_dns_dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8
28432024-03-30T05:44:59.721ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_pantry_a684a100-6031-4b97-8de3-41e15037fbd5
28442024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_pantry_f4d5df0a-748d-40ab-8235-7c927aae50ce
28452024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_47a9dbe8-5005-4fee-bb98-58944f67aa76
28462024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_22aed52b-11e9-4a7d-bebf-3356da024fc9
28472024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_0a93b238-dc4f-4234-98ad-cf545c1b8ea6
28482024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_nexus_f33d72e6-7ea4-4e54-8bfc-dc4233c612c4
28492024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_oximeter_b6a17b0f-0906-4eb9-94c3-528d000f68dc
28502024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_648e5f66-03f8-489e-a2bb-91cee8654a72
28512024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_nexus_cca30d41-f76f-4c5a-92f4-bb1cde1c17bd
28522024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_525c4f1a-5d69-4ecf-a43b-162fb1134523
28532024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_80d3c328-30b9-4b19-852b-d0002f908859
28542024-03-30T05:44:59.722ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2997 zone = oxz_crucible_06241231-d15c-4adb-ae5b-e8e364e35528
28552024-03-30T05:44:59.726ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
28562024-03-30T05:44:59.729ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
28572024-03-30T05:44:59.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 33893379 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52853 req_id = 3e82e629-fb4e-4785-8c27-47194697b0c4 response_code = 204 uri = /omicron-zones
28582024-03-30T05:44:59.765ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1230
28592024-03-30T05:44:59.765ZINFOSledAgent (RSS): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
28602024-03-30T05:44:59.765ZINFOSledAgent (RSS): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
28612024-03-30T05:44:59.768ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:677
28622024-03-30T05:44:59.772ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:685
28632024-03-30T05:44:59.818ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV1 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV1 { addresses: [ V4( Ipv4Network { addr: 10.151.1.174, prefix: 24, }, ), ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Network { addr: 0.0.0.0, prefix: 0, }, ), nexthop: 10.151.1.100, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, } file = sled-agent/src/rack_setup/service.rs:816
28642024-03-30T05:44:59.828ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28652024-03-30T05:44:59.949ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28662024-03-30T05:45:00.017ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28672024-03-30T05:45:00.191ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28682024-03-30T05:45:00.366ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28692024-03-30T05:45:00.451ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
28702024-03-30T05:45:00.587ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28712024-03-30T05:45:00.858ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28722024-03-30T05:45:01.067ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28732024-03-30T05:45:01.323ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28742024-03-30T05:45:01.817ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28752024-03-30T05:45:02.445ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
28762024-03-30T05:45:02.542ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28772024-03-30T05:45:03.403ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28782024-03-30T05:45:03.934ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28792024-03-30T05:45:04.444ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
28802024-03-30T05:45:04.764ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28812024-03-30T05:45:05.359ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28822024-03-30T05:45:06.240ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28832024-03-30T05:45:06.442ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
28842024-03-30T05:45:08.443ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/93b0cc91-dc62-481c-9ea1-425d29f058ae): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:619 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
28852024-03-30T05:45:08.455ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/98531190-dbea-48cf-b76b-c50c819faca1/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:873
28862024-03-30T05:45:10.463ZWARNSledAgent: failed to register as a metric producer with Nexus error = "Metric registration error: Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"e002f9fd-989d-40a4-b33b-a8415e04cb2a\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Sat, 30 Mar 2024 05:45:09 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"e002f9fd-989d-40a4-b33b-a8415e04cb2a\\" }" file = sled-agent/src/sled_agent.rs:1178 retry_after = 20.061729981s sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
28872024-03-30T05:45:11.336ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "51b41c50-860b-4403-b011-e7f74ead199f", "content-length": "124", "date": "Sat, 30 Mar 2024 05:45:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "51b41c50-860b-4403-b011-e7f74ead199f" } file = sled-agent/src/rack_setup/service.rs:873
28882024-03-30T05:45:12.492ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = 93b0cc91-dc62-481c-9ea1-425d29f058ae
28892024-03-30T05:45:12.832ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36942
28902024-03-30T05:45:12.834ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1361 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36942 req_id = d8edecdf-eda9-4298-9029-61fb32ccbafd response_code = 200 uri = /baseboard
28912024-03-30T05:45:13.503ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61863
28922024-03-30T05:45:13.504ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1039 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61863 req_id = ac0e60c8-4224-49aa-b5c3-54a0d4a26778 response_code = 200 uri = /network-bootstore-config
28932024-03-30T05:45:14.459ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34422
28942024-03-30T05:45:14.465ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:525 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 72eae3cb-c4dd-4f77-8b5b-946c1c2c7c41, kind: Service { id: dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8 }, name: Name("external-dns-dbd59cbf-fa3e-4fe2-b758-b2a136ae25a8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 195, 126])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9701e6c2-1d46-4a7d-b6e3-8b3125f74122, kind: Service { id: 57425cdc-844b-42e3-b45f-c8fb7871161f }, name: Name("external-dns-57425cdc-844b-42e3-b45f-c8fb7871161f"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 254, 213])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.1.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 21dc6cd8-6a7b-4ff0-9dd6-a2e7dc0c2a56, kind: Service { id: cca30d41-f76f-4c5a-92f4-bb1cde1c17bd }, name: Name("nexus-cca30d41-f76f-4c5a-92f4-bb1cde1c17bd"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 195])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 3630ad58-6099-41a3-8374-fe9c065b9ec4, kind: Service { id: f33d72e6-7ea4-4e54-8bfc-dc4233c612c4 }, name: Name("nexus-f33d72e6-7ea4-4e54-8bfc-dc4233c612c4"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 171, 116])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 591e0b7a-a507-4ecd-a23f-18a94999cc3e, kind: Service { id: 3b85f16b-9ecd-4867-ae56-7f8a1f492a4d }, name: Name("nexus-3b85f16b-9ecd-4867-ae56-7f8a1f492a4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 133, 194])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.2.0, prefix: 24 })), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
28952024-03-30T05:45:14.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28962024-03-30T05:45:14.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28972024-03-30T05:45:14.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28982024-03-30T05:45:14.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28992024-03-30T05:45:14.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
29002024-03-30T05:45:14.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:544 port = "opte0" rules = []
29012024-03-30T05:45:14.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 7557 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:34422 req_id = 89cf9bbc-87f0-4d35-ad08-47af3e741db3 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
29022024-03-30T05:45:14.468ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:883
29032024-03-30T05:45:14.691ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37621
29042024-03-30T05:45:14.692ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29052024-03-30T05:45:14.781ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 89474 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37621 req_id = 9ec194f5-0819-4af8-a70a-874a9daaee54 response_code = 200 uri = /inventory
29062024-03-30T05:45:14.782ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29072024-03-30T05:45:14.782ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/config/omicron-zones.json file = common/src/ledger.rs:177
29082024-03-30T05:45:14.783ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/config/omicron-zones.json file = common/src/ledger.rs:177
29092024-03-30T05:45:14.785ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3708 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37621 req_id = 10c19d31-6c6e-4bd6-bf11-718523716989 response_code = 200 uri = /omicron-zones
29102024-03-30T05:45:38.974ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40939
29112024-03-30T05:45:39.080ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 105889 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40939 req_id = 9dbe9981-afd9-422f-a065-530a750b1f76 response_code = 204 uri = /switch-ports
29122024-03-30T05:45:39.080ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46592
29132024-03-30T05:45:39.080ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 212 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40939 req_id = 00264ac6-4547-4bc8-884d-08ed3e0f46c5 response_code = 200 uri = /network-bootstore-config
29142024-03-30T05:45:39.109ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
29152024-03-30T05:45:39.110ZINFOSledAgent (bootstore): Writing ledger to /pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
29162024-03-30T05:45:39.111ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
29172024-03-30T05:45:39.112ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
29182024-03-30T05:45:39.112ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3786 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40939 req_id = b2cee4fe-7a4e-458e-a323-68f730a1f0d3 response_code = 204 uri = /network-bootstore-config
29192024-03-30T05:45:39.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59216
29202024-03-30T05:45:39.142ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 61652 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46592 req_id = c3bee200-cc6b-4dee-a04c-2c434e149428 response_code = 204 uri = /switch-ports
29212024-03-30T05:45:39.146ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 636 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46592 req_id = 5cf4cddb-53f2-49ca-b924-37fa60f2a4be response_code = 200 uri = /network-bootstore-config
29222024-03-30T05:45:39.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 52927 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59216 req_id = d7d52828-68e5-40ed-ba8f-7c5b1e6ccb11 response_code = 204 uri = /switch-ports
29232024-03-30T05:45:39.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59216 req_id = c959bfde-5c7b-43f7-b7b8-7c18236970d1 response_code = 200 uri = /network-bootstore-config
29242024-03-30T05:45:42.870ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63431
29252024-03-30T05:45:42.870ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 70 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:63431 req_id = f6363e42-d012-4efc-b3dc-12b97d1b47fc response_code = 200 uri = /baseboard
29262024-03-30T05:46:00.602ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::d]:56676
29272024-03-30T05:46:00.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 9353 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = e58bbf94-06b1-46f9-a6a5-6da03eb3e6fa response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
29282024-03-30T05:46:09.141ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51346
29292024-03-30T05:46:09.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 51022 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51346 req_id = b8d11dc1-1583-4682-b0e1-c4b900a4278a response_code = 204 uri = /switch-ports
29302024-03-30T05:46:09.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 977 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51346 req_id = 73da25f5-4d9b-42d4-bbb2-97b0fac04fdc response_code = 200 uri = /network-bootstore-config
29312024-03-30T05:46:09.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57762
29322024-03-30T05:46:09.306ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45922 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57762 req_id = 6556da29-14e6-4302-9e92-1644a9d47d50 response_code = 204 uri = /switch-ports
29332024-03-30T05:46:09.307ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57762 req_id = aedf4b1e-41dd-40f2-ac16-ed07d1b42948 response_code = 200 uri = /network-bootstore-config
29342024-03-30T05:46:09.483ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62543
29352024-03-30T05:46:09.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45508 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62543 req_id = 14b1c8ee-150e-4ea3-b62a-65c104f2eede response_code = 204 uri = /switch-ports
29362024-03-30T05:46:09.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 757 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62543 req_id = 8d21d3f3-d90f-4262-9056-041b5c9be6c6 response_code = 200 uri = /network-bootstore-config
29372024-03-30T05:46:12.912ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42016
29382024-03-30T05:46:12.912ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 88 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42016 req_id = 426d25e5-b5fc-4d72-8b3e-9612aecf29ac response_code = 200 uri = /baseboard
29392024-03-30T05:46:30.607ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1260 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 79942226-4c49-4276-acbd-b2112dfe98e0 response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
29402024-03-30T05:46:38.820ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39353
29412024-03-30T05:46:38.869ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 49130 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39353 req_id = 7420616d-037c-4083-aa46-16995ceb81a1 response_code = 204 uri = /switch-ports
29422024-03-30T05:46:38.870ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 797 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39353 req_id = 3a5dbfc8-31ca-4c8d-b3e9-380129df543c response_code = 200 uri = /network-bootstore-config
29432024-03-30T05:46:38.922ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63412
29442024-03-30T05:46:38.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44265 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63412 req_id = 9a1c674a-fc82-4abc-9b68-cf7fcd7804cb response_code = 204 uri = /switch-ports
29452024-03-30T05:46:38.966ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54098
29462024-03-30T05:46:39.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43183 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54098 req_id = 353e37d6-261e-4b51-905f-789b54871459 response_code = 204 uri = /switch-ports
29472024-03-30T05:46:39.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63412 req_id = 2c901df2-520b-4ef7-937e-6537d0f71b16 response_code = 200 uri = /network-bootstore-config
29482024-03-30T05:46:39.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54098 req_id = ee6bf5af-3611-41f9-b0a5-ae83e6222b39 response_code = 200 uri = /network-bootstore-config
29492024-03-30T05:46:42.949ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48500
29502024-03-30T05:46:42.949ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 73 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:48500 req_id = d4fd3a5e-67d9-4513-a709-3adeddc79da9 response_code = 200 uri = /baseboard
29512024-03-30T05:47:00.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 570 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 511f9ba0-1776-4eea-9c3a-8d4623837f0e response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
29522024-03-30T05:47:09.070ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54067
29532024-03-30T05:47:09.116ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46125 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54067 req_id = d6a934d9-b6e6-43ad-9990-0bbe133822bb response_code = 204 uri = /switch-ports
29542024-03-30T05:47:09.116ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54067 req_id = fb596565-635d-4f53-aa49-8bc79822b04e response_code = 200 uri = /network-bootstore-config
29552024-03-30T05:47:09.126ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60917
29562024-03-30T05:47:09.171ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45240 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60917 req_id = f59a5ef5-7763-4a20-b794-a09fd44b2960 response_code = 204 uri = /switch-ports
29572024-03-30T05:47:09.171ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60917 req_id = f61bb8bc-05cf-4f98-9974-9f9bbfc0d318 response_code = 200 uri = /network-bootstore-config
29582024-03-30T05:47:09.183ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59631
29592024-03-30T05:47:09.227ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43387 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59631 req_id = aef18d9b-b63b-4fa9-8334-2533cfec3218 response_code = 204 uri = /switch-ports
29602024-03-30T05:47:09.227ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59631 req_id = 9c894446-905a-474e-a5eb-ece07cb58e9d response_code = 200 uri = /network-bootstore-config
29612024-03-30T05:47:12.985ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58412
29622024-03-30T05:47:12.986ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 48 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:58412 req_id = df1a53d9-e0bf-4bd1-9ef3-2409ba4e94c4 response_code = 200 uri = /baseboard
29632024-03-30T05:47:30.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 560 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 080bbd47-cf62-4cce-b84f-673076edcc27 response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
29642024-03-30T05:47:38.921ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50481
29652024-03-30T05:47:38.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 48260 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50481 req_id = 88174522-8692-4834-994b-01ce015e34d4 response_code = 204 uri = /switch-ports
29662024-03-30T05:47:38.970ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50481 req_id = 4bbb829e-b583-4f66-ae47-1200a7a8bb8f response_code = 200 uri = /network-bootstore-config
29672024-03-30T05:47:39.122ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52498
29682024-03-30T05:47:39.168ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46586 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52498 req_id = 47b4247e-4bfe-4633-b414-3554402fc309 response_code = 204 uri = /switch-ports
29692024-03-30T05:47:39.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 657 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52498 req_id = 14040480-69e3-4fdc-8729-98f1737e8288 response_code = 200 uri = /network-bootstore-config
29702024-03-30T05:47:39.184ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42287
29712024-03-30T05:47:39.231ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47429 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:42287 req_id = 2375fd14-54ac-4e47-8432-73509000ae90 response_code = 204 uri = /switch-ports
29722024-03-30T05:47:39.232ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42287 req_id = 53313441-9d91-4e64-ab42-45cc3f65f154 response_code = 200 uri = /network-bootstore-config
29732024-03-30T05:47:43.021ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42972
29742024-03-30T05:47:43.021ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 114 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42972 req_id = 6e117d56-928f-443d-bdb6-ab5e23bbfb0b response_code = 200 uri = /baseboard
29752024-03-30T05:48:00.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 553 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 88bb69ce-d103-45d8-9f4b-0c356fd8ccdb response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
29762024-03-30T05:48:05.967ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:801
29772024-03-30T05:48:09.185ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49048
29782024-03-30T05:48:09.232ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46912 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:49048 req_id = 33307c6b-3ebb-4a2f-81d7-147eedd85e2a response_code = 204 uri = /switch-ports
29792024-03-30T05:48:09.232ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49048 req_id = b40f0bad-60ea-4170-bffd-e27165e6307e response_code = 200 uri = /network-bootstore-config
29802024-03-30T05:48:09.242ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64697
29812024-03-30T05:48:09.285ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43207 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64697 req_id = 5111306f-3fdf-40ba-8d85-49c3317a4801 response_code = 204 uri = /switch-ports
29822024-03-30T05:48:09.286ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64697 req_id = 38e04c74-276f-4629-ad12-f83b7e819061 response_code = 200 uri = /network-bootstore-config
29832024-03-30T05:48:09.854ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55364
29842024-03-30T05:48:09.902ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 48147 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:55364 req_id = ed42fa22-2e1f-41ea-bbd7-4fa536260f6b response_code = 204 uri = /switch-ports
29852024-03-30T05:48:09.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55364 req_id = db2160be-9277-44cf-9df5-53f540a2c948 response_code = 200 uri = /network-bootstore-config
29862024-03-30T05:48:13.053ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54882
29872024-03-30T05:48:13.053ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 62 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:54882 req_id = 73cdf5e0-7947-44dc-a047-b98537053f19 response_code = 200 uri = /baseboard
29882024-03-30T05:48:30.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 504 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = fef2e675-e0c7-47e8-8a99-54038e0505db response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
29892024-03-30T05:48:38.927ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58537
29902024-03-30T05:48:38.975ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 48165 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58537 req_id = e1bf3a36-f417-4099-b24f-a60b7ae0a4ff response_code = 204 uri = /switch-ports
29912024-03-30T05:48:38.975ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61628
29922024-03-30T05:48:38.979ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3437 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58537 req_id = 879d549c-6119-4606-b77d-ad79466437c1 response_code = 200 uri = /network-bootstore-config
29932024-03-30T05:48:39.021ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46638
29942024-03-30T05:48:39.023ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 48146 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61628 req_id = 5437e840-a4c1-4897-a92c-6ed4abcca6d0 response_code = 204 uri = /switch-ports
29952024-03-30T05:48:39.027ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 569 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61628 req_id = f0e51c58-5096-4eab-bbd4-ff1521dd2cc5 response_code = 200 uri = /network-bootstore-config
29962024-03-30T05:48:39.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 51515 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46638 req_id = 90c807e8-a06b-49e1-81aa-3bf199a30a55 response_code = 204 uri = /switch-ports
29972024-03-30T05:48:39.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46638 req_id = 0603d7d0-b28a-41cd-903d-b387d8fdb061 response_code = 200 uri = /network-bootstore-config
29982024-03-30T05:48:43.086ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44483
29992024-03-30T05:48:43.087ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 64 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:44483 req_id = 46fa546e-b481-41a5-bab4-429e793f7d45 response_code = 200 uri = /baseboard
30002024-03-30T05:49:00.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 505 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 0b139f4f-c337-4465-903a-3d37d4390557 response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
30012024-03-30T05:49:09.014ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57901
30022024-03-30T05:49:09.059ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45630 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57901 req_id = c4baaaed-ff4d-4eee-8323-be286ccc1430 response_code = 204 uri = /switch-ports
30032024-03-30T05:49:09.060ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57901 req_id = fbe3d5ff-e1ef-4810-8181-be8943d5027e response_code = 200 uri = /network-bootstore-config
30042024-03-30T05:49:09.172ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50138
30052024-03-30T05:49:09.219ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 46904 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:50138 req_id = 6e4ebabe-ff03-444d-b393-e739b45967ce response_code = 204 uri = /switch-ports
30062024-03-30T05:49:09.220ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50138 req_id = 32b20fa5-4e70-4ba3-ba14-0d6f96951bc4 response_code = 200 uri = /network-bootstore-config
30072024-03-30T05:49:09.488ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45132
30082024-03-30T05:49:09.533ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44788 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45132 req_id = d354af8e-26d8-499d-991c-337536600860 response_code = 204 uri = /switch-ports
30092024-03-30T05:49:09.533ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45132 req_id = a9376f83-21aa-4c39-bdd2-14749da210e9 response_code = 200 uri = /network-bootstore-config
30102024-03-30T05:49:13.121ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55306
30112024-03-30T05:49:13.121ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 80 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55306 req_id = 567e3145-d679-445f-a51f-9aa9894fd09f response_code = 200 uri = /baseboard
30122024-03-30T05:49:30.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 496 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 7dbddfbb-4054-4b9b-ae7a-f18c34c75c6b response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
30132024-03-30T05:49:38.839ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33001
30142024-03-30T05:49:38.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47389 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33001 req_id = fd9161b0-ac26-4b75-826e-1e0181ead097 response_code = 204 uri = /switch-ports
30152024-03-30T05:49:38.887ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33001 req_id = 803c0665-3bfa-4a0c-811b-b428947c844a response_code = 200 uri = /network-bootstore-config
30162024-03-30T05:49:38.946ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37035
30172024-03-30T05:49:38.987ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 41445 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37035 req_id = 20238f55-b44d-4e86-9c54-a81783980e3a response_code = 204 uri = /switch-ports
30182024-03-30T05:49:38.988ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 586 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37035 req_id = 8cf671c5-927b-49be-ade1-8b94b2b6f631 response_code = 200 uri = /network-bootstore-config
30192024-03-30T05:49:39.043ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59550
30202024-03-30T05:49:39.087ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44267 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:59550 req_id = 9ead0c95-774d-4628-be1d-a07c212a44b6 response_code = 204 uri = /switch-ports
30212024-03-30T05:49:39.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59550 req_id = 6c6b0503-92f1-4143-abc0-c8ef1442ec76 response_code = 200 uri = /network-bootstore-config
30222024-03-30T05:49:43.153ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40701
30232024-03-30T05:49:43.154ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 90 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:40701 req_id = aeea4a0f-0183-405a-a16c-194dee72a26c response_code = 200 uri = /baseboard
30242024-03-30T05:50:00.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 475 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 44a85251-d6a0-48c9-bf3b-80e686914e53 response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
30252024-03-30T05:50:09.205ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33829
30262024-03-30T05:50:09.248ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 42663 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:33829 req_id = e86e281a-d5ad-4a39-a980-7373e6268436 response_code = 204 uri = /switch-ports
30272024-03-30T05:50:09.248ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43007
30282024-03-30T05:50:09.251ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3395 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33829 req_id = 9b792743-32b5-4a44-98c8-39ae6d3b4de9 response_code = 200 uri = /network-bootstore-config
30292024-03-30T05:50:09.289ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 40916 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43007 req_id = f415c1bc-a38b-4195-ad05-b8b1c6ccc3d3 response_code = 204 uri = /switch-ports
30302024-03-30T05:50:09.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 689 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43007 req_id = 33aed7cd-cb57-4d01-a75d-3f03f84ff8da response_code = 200 uri = /network-bootstore-config
30312024-03-30T05:50:09.299ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36866
30322024-03-30T05:50:09.341ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 41681 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36866 req_id = e817e582-9010-4fa9-86d2-3a80a476d05c response_code = 204 uri = /switch-ports
30332024-03-30T05:50:09.341ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36866 req_id = 6a902c27-989a-4f7d-915c-089b9bfe834c response_code = 200 uri = /network-bootstore-config
30342024-03-30T05:50:13.186ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43719
30352024-03-30T05:50:13.186ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 136 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:43719 req_id = e9b85c2b-35db-4d07-9e10-2ac0c08b4e2b response_code = 200 uri = /baseboard
30362024-03-30T05:50:30.612ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1289 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 7ee07537-df6d-4398-aa32-0a7d36df33cf response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
30372024-03-30T05:50:38.917ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64545
30382024-03-30T05:50:38.963ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45543 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64545 req_id = 6a16d928-9845-4001-9f45-7887355cab3c response_code = 204 uri = /switch-ports
30392024-03-30T05:50:38.963ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57860
30402024-03-30T05:50:38.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3694 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64545 req_id = 4bfa7251-4852-4ae8-b7aa-ca4f9775cbe8 response_code = 200 uri = /network-bootstore-config
30412024-03-30T05:50:39.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45925 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57860 req_id = f5cbbaf7-c3a9-40db-b59b-9aefc5db787b response_code = 204 uri = /switch-ports
30422024-03-30T05:50:39.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57860 req_id = 1c8e80f7-32b4-4134-8fe3-2b1a6655febb response_code = 200 uri = /network-bootstore-config
30432024-03-30T05:50:39.039ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51031
30442024-03-30T05:50:39.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44869 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:51031 req_id = c6c27a70-3226-4be1-b9f2-b399cbb99d68 response_code = 204 uri = /switch-ports
30452024-03-30T05:50:39.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51031 req_id = 86a159a1-d3c6-4adb-95aa-d5b1ae53b435 response_code = 200 uri = /network-bootstore-config
30462024-03-30T05:50:43.219ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57819
30472024-03-30T05:50:43.219ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 78 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57819 req_id = 4469fd9b-e5d4-475b-9609-fc71e130e384 response_code = 200 uri = /baseboard
30482024-03-30T05:51:00.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 490 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 2b9e90dd-3055-465e-8505-039a81eeb344 response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
30492024-03-30T05:51:09.161ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36512
30502024-03-30T05:51:09.206ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44880 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:36512 req_id = 6023ecda-ca29-4b69-869f-73bb22816f16 response_code = 204 uri = /switch-ports
30512024-03-30T05:51:09.207ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 592 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36512 req_id = 4ea1f4f7-c772-4f44-9a7e-3d62a463fc65 response_code = 200 uri = /network-bootstore-config
30522024-03-30T05:51:09.255ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48395
30532024-03-30T05:51:09.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45860 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48395 req_id = aeedcda3-9a72-4eda-a150-667e74209ffd response_code = 204 uri = /switch-ports
30542024-03-30T05:51:09.301ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41833
30552024-03-30T05:51:09.347ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 45553 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41833 req_id = 1d861eda-db97-42f0-b06c-215a53f687d7 response_code = 204 uri = /switch-ports
30562024-03-30T05:51:09.347ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48395 req_id = ba6f360a-47d5-434c-bbb2-6cee2fc595e2 response_code = 200 uri = /network-bootstore-config
30572024-03-30T05:51:09.347ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41833 req_id = c30266d4-cfce-47f7-809d-015efe8b30b0 response_code = 200 uri = /network-bootstore-config
30582024-03-30T05:51:13.251ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46485
30592024-03-30T05:51:13.251ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 72 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:46485 req_id = d32d8be6-a0b9-4c3c-81ef-ba39093dd6df response_code = 200 uri = /baseboard
30602024-03-30T05:51:30.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 535 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 7d0b408a-2797-45e5-b630-345ac029d97d response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
30612024-03-30T05:51:38.828ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41652
30622024-03-30T05:51:38.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 47100 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41652 req_id = 8fac975d-dd4f-49f1-8dc1-02516313f915 response_code = 204 uri = /switch-ports
30632024-03-30T05:51:38.875ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45018
30642024-03-30T05:51:38.879ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 3796 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41652 req_id = 5655250e-5aec-425f-8b2f-dc4f46dd1e7c response_code = 200 uri = /network-bootstore-config
30652024-03-30T05:51:38.927ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 51260 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:45018 req_id = 271dde93-9ea5-4527-98fb-f109c7baac98 response_code = 204 uri = /switch-ports
30662024-03-30T05:51:38.927ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 623 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45018 req_id = cea769d0-72ef-4c4a-b237-77b6bfb47058 response_code = 200 uri = /network-bootstore-config
30672024-03-30T05:51:38.936ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63779
30682024-03-30T05:51:38.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44248 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63779 req_id = ceec1e54-8c97-4822-8363-06d9edeeed2c response_code = 204 uri = /switch-ports
30692024-03-30T05:51:38.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63779 req_id = c650020e-65ca-4652-8ac8-0f57a6cd5d8e response_code = 200 uri = /network-bootstore-config
30702024-03-30T05:51:43.284ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55029
30712024-03-30T05:51:43.284ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 711 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55029 req_id = c23242de-6530-4240-a308-19b722b2e467 response_code = 200 uri = /baseboard
30722024-03-30T05:51:50.731ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39459
30732024-03-30T05:51:50.743ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:521 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8, kind: Instance { id: daec6b21-c8aa-4afd-a556-e93742ed7c15 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 119, 207])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(16749977), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8, kind: Instance { id: daec6b21-c8aa-4afd-a556-e93742ed7c15 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 119, 207])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(16749977), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8, kind: Instance { id: daec6b21-c8aa-4afd-a556-e93742ed7c15 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 119, 207])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(16749977), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(16749977))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8, kind: Instance { id: daec6b21-c8aa-4afd-a556-e93742ed7c15 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 119, 207])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(16749977), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-f714748b0f3", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 221292c7-c1ce-449c-9a71-878f99132105, read_only_parent: Some(Volume { block_size: 512, id: 7210a1d9-d22d-49a7-87cd-02b628f37500, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 44eb95f9-0471-4a3f-af93-02e8ebe57571, key: Some("9pYb0mSs5cFWpa0MYvWpQcvVydm3Ueo3f/zh/alOoc4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19002", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::15]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 221292c7-c1ce-449c-9a71-878f99132105, key: Some("SsgD9wq41gdRqKScyoI1RdX5djH0z6OYhPm0Sr76aJE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::12]:19000", "[fd00:1122:3344:101::14]:19001"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABEeS5+WH5YAAB5Ln5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAARXkuflh+WAAAeS5+WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJkYWVjNmIyMS1jOGFhLTRhZmQtYTU1Ni1lOTM3NDJlZDdjMTUiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJQmtvVFJpNndHQ0VUMHhyYXQ5RXoyeUtVZHdzdVhQUzZ1dWVhNk5CT3habCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 instance_runtime = InstanceRuntimeState { propolis_id: Some(b6608e45-b185-4760-b19c-f24aceaaafd2), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-03-30T05:51:50.355854109Z } metadata = InstanceMetadata { silo_id: e48373c8-b487-4cd1-adc6-a2c22d299e4c, project_id: 9bf892ed-1a58-4dcc-9342-5b6618092742 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = b6608e45-b185-4760-b19c-f24aceaaafd2 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-03-30T05:51:50.329873Z }
30742024-03-30T05:51:50.744ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:555 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30752024-03-30T05:51:50.744ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:938 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 propolis_id = b6608e45-b185-4760-b19c-f24aceaaafd2 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8, kind: Instance { id: daec6b21-c8aa-4afd-a556-e93742ed7c15 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 119, 207])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(16749977), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8, kind: Instance { id: daec6b21-c8aa-4afd-a556-e93742ed7c15 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 119, 207])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(16749977), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8, kind: Instance { id: daec6b21-c8aa-4afd-a556-e93742ed7c15 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 119, 207])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(16749977), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(16749977))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8, kind: Instance { id: daec6b21-c8aa-4afd-a556-e93742ed7c15 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 119, 207])), subnet: V4(Ipv4Net(Ipv4Network { addr: 172.30.0.0, prefix: 22 })), vni: Vni(16749977), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-f714748b0f3", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 221292c7-c1ce-449c-9a71-878f99132105, read_only_parent: Some(Volume { block_size: 512, id: 7210a1d9-d22d-49a7-87cd-02b628f37500, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 44eb95f9-0471-4a3f-af93-02e8ebe57571, key: Some("9pYb0mSs5cFWpa0MYvWpQcvVydm3Ueo3f/zh/alOoc4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19002", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::15]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 221292c7-c1ce-449c-9a71-878f99132105, key: Some("SsgD9wq41gdRqKScyoI1RdX5djH0z6OYhPm0Sr76aJE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::12]:19000", "[fd00:1122:3344:101::14]:19001"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABEeS5+WH5YAAB5Ln5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAARXkuflh+WAAAeS5+WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJkYWVjNmIyMS1jOGFhLTRhZmQtYTU1Ni1lOTM3NDJlZDdjMTUiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJQmtvVFJpNndHQ0VUMHhyYXQ5RXoyeUtVZHdzdVhQUzZ1dWVhNk5CT3habCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(b6608e45-b185-4760-b19c-f24aceaaafd2), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-03-30T05:51:50.355854109Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-03-30T05:51:50.329873Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
30762024-03-30T05:51:50.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 13034 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39459 req_id = fbd154a7-4426-4260-b38c-4a7be595be84 response_code = 200 uri = /instances/daec6b21-c8aa-4afd-a556-e93742ed7c15
30772024-03-30T05:51:50.833ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59534
30782024-03-30T05:51:50.835ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 1372 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59534 req_id = 196ec3a0-18ba-4d4c-9046-fee6e8c08c00 response_code = 200 uri = /instances/daec6b21-c8aa-4afd-a556-e93742ed7c15/state
30792024-03-30T05:51:50.859ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:394 port = Port { inner: PortInner { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 250, 119, 207]), slot: 0, vni: Vni { inner: 16749977 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 }, vnic: "vopte6" } }
30802024-03-30T05:51:50.860ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
30812024-03-30T05:51:50.894ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2 file = illumos-utils/src/zone.rs:315 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30822024-03-30T05:51:50.914ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2 file = illumos-utils/src/zone.rs:354 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30832024-03-30T05:51:52.936ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1:0"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::a]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30842024-03-30T05:51:52.937ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30852024-03-30T05:51:53.128ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 60.49588ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30862024-03-30T05:51:53.211ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 124.09068ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30872024-03-30T05:51:53.355ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 174.781607ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30882024-03-30T05:51:53.540ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 267.855103ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30892024-03-30T05:51:53.817ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 859.613252ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30902024-03-30T05:51:54.688ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 1.403540649s file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30912024-03-30T05:51:56.103ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 1.013682736s file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30922024-03-30T05:51:57.127ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 615.82741ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 zone = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
30932024-03-30T05:51:57.773ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2 file = sled-agent/src/instance.rs:1423 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30942024-03-30T05:51:57.783ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 44.676269ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30952024-03-30T05:51:57.840ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 106.480349ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30962024-03-30T05:51:57.957ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 229.802988ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30972024-03-30T05:51:58.197ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2") failed: Property not found. retry in 583.68864ms file = illumos-utils/src/svc.rs:36 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30982024-03-30T05:51:58.791ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1432 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
30992024-03-30T05:51:58.827ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1446 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31002024-03-30T05:51:58.833ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABEeS5+WH5YAAB5Ln5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAARXkuflh+WAAAeS5+WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJkYWVjNmIyMS1jOGFhLTRhZmQtYTU1Ni1lOTM3NDJlZDdjMTUiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJQmtvVFJpNndHQ0VUMHhyYXQ5RXoyeUtVZHdzdVhQUzZ1dWVhNk5CT3habCJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-f714748b0f3", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 221292c7-c1ce-449c-9a71-878f99132105, read_only_parent: Some(Volume { block_size: 512, id: 7210a1d9-d22d-49a7-87cd-02b628f37500, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 44eb95f9-0471-4a3f-af93-02e8ebe57571, key: Some("9pYb0mSs5cFWpa0MYvWpQcvVydm3Ueo3f/zh/alOoc4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19002", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::15]:19002"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 221292c7-c1ce-449c-9a71-878f99132105, key: Some("SsgD9wq41gdRqKScyoI1RdX5djH0z6OYhPm0Sr76aJE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::12]:19000", "[fd00:1122:3344:101::14]:19001"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "vopte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: daec6b21-c8aa-4afd-a556-e93742ed7c15, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 9bf892ed-1a58-4dcc-9342-5b6618092742, silo_id: e48373c8-b487-4cd1-adc6-a2c22d299e4c }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:685 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31012024-03-30T05:51:58.998ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:687 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31022024-03-30T05:51:58.998ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-03-30T05:51:58.998625290Z } file = sled-agent/src/instance.rs:569 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31032024-03-30T05:51:58.998ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 new_instance_state = InstanceRuntimeState { propolis_id: Some(b6608e45-b185-4760-b19c-f24aceaaafd2), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-03-30T05:51:50.355854109Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-03-30T05:51:58.998625290Z } propolis_id = b6608e45-b185-4760-b19c-f24aceaaafd2
31042024-03-30T05:51:58.998ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(b6608e45-b185-4760-b19c-f24aceaaafd2), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-03-30T05:51:50.355854109Z }, propolis_id: b6608e45-b185-4760-b19c-f24aceaaafd2, vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-03-30T05:51:58.998625290Z } }
31052024-03-30T05:51:59.058ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-03-30T05:51:59.058814279Z } file = sled-agent/src/instance.rs:569 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31062024-03-30T05:51:59.058ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 new_instance_state = InstanceRuntimeState { propolis_id: Some(b6608e45-b185-4760-b19c-f24aceaaafd2), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-03-30T05:51:50.355854109Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-03-30T05:51:59.058814279Z } propolis_id = b6608e45-b185-4760-b19c-f24aceaaafd2
31072024-03-30T05:51:59.058ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(b6608e45-b185-4760-b19c-f24aceaaafd2), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-03-30T05:51:50.355854109Z }, propolis_id: b6608e45-b185-4760-b19c-f24aceaaafd2, vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-03-30T05:51:59.058814279Z } }
31082024-03-30T05:52:00.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 558 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::d]:56676 req_id = 5b3c2898-85ff-41ff-b20e-3b016e086931 response_code = 200 uri = /metrics/collect/93b0cc91-dc62-481c-9ea1-425d29f058ae
31092024-03-30T05:52:07.571ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40666
31102024-03-30T05:52:07.572ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 626 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40666 req_id = 7ca76ef8-dc4c-4684-83a9-41dce19229aa response_code = 200 uri = /instances/daec6b21-c8aa-4afd-a556-e93742ed7c15/state
31112024-03-30T05:52:07.572ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-03-30T05:52:07.572484614Z } file = sled-agent/src/instance.rs:569 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31122024-03-30T05:52:07.572ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 new_instance_state = InstanceRuntimeState { propolis_id: Some(b6608e45-b185-4760-b19c-f24aceaaafd2), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-03-30T05:51:50.355854109Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-03-30T05:52:07.572484614Z } propolis_id = b6608e45-b185-4760-b19c-f24aceaaafd2
31132024-03-30T05:52:07.572ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(b6608e45-b185-4760-b19c-f24aceaaafd2), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-03-30T05:51:50.355854109Z }, propolis_id: b6608e45-b185-4760-b19c-f24aceaaafd2, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-03-30T05:52:07.572484614Z } }
31142024-03-30T05:52:07.594ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-03-30T05:52:07.594394020Z } file = sled-agent/src/instance.rs:569 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31152024-03-30T05:52:07.594ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-03-30T05:52:07.594394020Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-03-30T05:52:07.594394020Z } propolis_id = b6608e45-b185-4760-b19c-f24aceaaafd2
31162024-03-30T05:52:07.594ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:606 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31172024-03-30T05:52:07.595ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
31182024-03-30T05:52:07.597ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
31192024-03-30T05:52:07.597ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/eaeb58f9-7756-4551-8a4b-dc00787e5906/debug/bundle/zone", "/pool/int/f076b946-15f4-4270-9a34-aa6e75797fbb/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/82347231-68ab-4934-8007-b28d9a3a3777/crypt/debug", "/pool/ext/28899eb6-914e-44b6-b77e-9d82d92186d6/crypt/debug", "/pool/ext/6de8542c-123a-4abc-9a88-e72f1063a9f4/crypt/debug", "/pool/ext/e43353e3-7e66-4e6f-bd56-19f8143ee8b5/crypt/debug", "/pool/ext/5c3019ca-328c-47ab-9982-06e092199db8/crypt/debug", "/pool/ext/3c92b6bd-8180-45a3-9123-a99ddc649159/crypt/debug", "/pool/ext/ea19d3d4-e058-46fb-981b-9ccb255f563c/crypt/debug", "/pool/ext/e22df0ad-bbf8-4902-93d6-e67ff23c3234/crypt/debug", "/pool/ext/c7e0e489-6c2b-472e-8a1f-efe83bb548dd/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:437 zone_name = oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2
31202024-03-30T05:52:08.197ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1236 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2", bundle_id: 2a5eb9ef-7a71-4103-b991-5f0a32187f1d }, time_created: 2024-03-30T05:52:07.598294710Z, version: 0, cause: TerminatedInstance }
31212024-03-30T05:52:08.197ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_b6608e45-b185-4760-b19c-f24aceaaafd2 file = sled-agent/src/instance.rs:780 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31222024-03-30T05:52:09.069ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42902
31232024-03-30T05:52:09.112ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 43062 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42902 req_id = 0f55b517-5a9c-4f00-a1bf-044fb3962297 response_code = 204 uri = /switch-ports
31242024-03-30T05:52:09.112ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42902 req_id = b8cd97ee-82f5-4e15-96fd-896311144a11 response_code = 200 uri = /network-bootstore-config
31252024-03-30T05:52:09.191ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46068
31262024-03-30T05:52:09.235ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 44030 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46068 req_id = bb297b8a-3ff6-40f3-8bf0-19620fede39b response_code = 204 uri = /switch-ports
31272024-03-30T05:52:09.236ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 885 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46068 req_id = 30fe8926-0533-4cb7-ad20-fcc56fd8b59c response_code = 200 uri = /network-bootstore-config
31282024-03-30T05:52:09.293ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47720
31292024-03-30T05:52:09.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 49408 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47720 req_id = 9861bf28-e308-4368-8dd1-dad3742ab7fd response_code = 204 uri = /switch-ports
31302024-03-30T05:52:09.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47720 req_id = 791b2f8a-151d-4341-8099-c189e4a3507c response_code = 200 uri = /network-bootstore-config
31312024-03-30T05:52:09.948ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:269 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15
31322024-03-30T05:52:09.972ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-03-30T05:52:07.594394020Z }, propolis_id: b6608e45-b185-4760-b19c-f24aceaaafd2, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-03-30T05:52:09.972458882Z } }
31332024-03-30T05:52:10.049ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:765 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57877
31342024-03-30T05:52:10.054ZWARNSledAgent (PortManager): unset_virtual_nic_host unimplmented file = illumos-utils/src/opte/port_manager.rs:620
31352024-03-30T05:52:10.054ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/29ae98d/dropshot/src/server.rs:849 latency_us = 2530 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:57877 req_id = 397a391e-6153-4fda-b162-819fd36e4271 response_code = 204 uri = /v2p/57abfa80-97d5-4f38-b9f3-4d7ea6b36aa8
31362024-03-30T05:52:10.282ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-03-30T05:52:07.594394020Z }, propolis_id: b6608e45-b185-4760-b19c-f24aceaaafd2, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-03-30T05:52:09.972458882Z } }
31372024-03-30T05:52:10.284ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:723 instance_id = daec6b21-c8aa-4afd-a556-e93742ed7c15