Max level shown:
1 [ May 30 04:07:46 Enabled. ]
2 [ May 30 04:07:46 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ May 30 04:07:46 Method "start" exited with status 0. ]
4 [ May 30 04:07:46 Rereading configuration. ]
5 [ May 30 04:07:46 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-05-30T04:07:46.386ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:163
82024-05-30T04:07:46.386ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-05-30T04:07:46.439ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-05-30T04:07:46.439ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-05-30T04:07:46.555ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-05-30T04:07:46.564ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-05-30T04:07:46.572ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-05-30T04:07:47.175ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:a02a::, len: 64 }
152024-05-30T04:07:47.313ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:119
162024-05-30T04:07:47.313ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:131
172024-05-30T04:07:47.313ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:141
182024-05-30T04:07:47.313ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:162 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-05-30T04:07:47.313ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:506
202024-05-30T04:07:47.313ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:625
212024-05-30T04:07:47.317ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:535
222024-05-30T04:07:47.317ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:674
232024-05-30T04:07:47.317ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:176
242024-05-30T04:07:47.317ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:733
252024-05-30T04:07:47.317ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_0.vdev
262024-05-30T04:07:47.317ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = None
272024-05-30T04:07:47.317ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }), tx: ... } file = sled-storage/src/manager.rs:381
282024-05-30T04:07:47.317ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-05-30T04:07:47.317ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-05-30T04:07:47.328ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:304
312024-05-30T04:07:47.328ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = 75663f09-f516-40fc-852b-77e4a30319f5 (zpool)
322024-05-30T04:07:47.425ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal }
332024-05-30T04:07:47.628ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
342024-05-30T04:07:47.628ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_1.vdev
352024-05-30T04:07:47.628ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-05-30T04:07:47.628ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-05-30T04:07:47.628ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }), tx: ... } file = sled-storage/src/manager.rs:381
382024-05-30T04:07:47.628ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-05-30T04:07:47.628ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-05-30T04:07:47.637ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:304
412024-05-30T04:07:47.637ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = 804b9788-bd41-4d87-a91b-e568478fa644 (zpool)
422024-05-30T04:07:47.729ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal }
432024-05-30T04:07:47.930ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_0.vdev
442024-05-30T04:07:47.930ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
452024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
462024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
472024-05-30T04:07:47.930ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }), tx: ... } file = sled-storage/src/manager.rs:381
482024-05-30T04:07:47.930ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-05-30T04:07:47.930ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_1.vdev
502024-05-30T04:07:47.930ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
512024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
522024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
532024-05-30T04:07:47.930ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }), tx: ... } file = sled-storage/src/manager.rs:381
542024-05-30T04:07:47.930ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-05-30T04:07:47.930ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_2.vdev
562024-05-30T04:07:47.930ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }), tx: ... } file = sled-storage/src/manager.rs:381
572024-05-30T04:07:47.930ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
582024-05-30T04:07:47.930ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
602024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
612024-05-30T04:07:47.930ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_3.vdev
622024-05-30T04:07:47.930ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
632024-05-30T04:07:47.930ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }), tx: ... } file = sled-storage/src/manager.rs:381
642024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
652024-05-30T04:07:47.930ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
662024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
672024-05-30T04:07:47.930ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_4.vdev
682024-05-30T04:07:47.930ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }), tx: ... } file = sled-storage/src/manager.rs:381
692024-05-30T04:07:47.930ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
702024-05-30T04:07:47.930ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
712024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
722024-05-30T04:07:47.930ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
732024-05-30T04:07:47.930ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_5.vdev
742024-05-30T04:07:47.930ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }), tx: ... } file = sled-storage/src/manager.rs:381
752024-05-30T04:07:47.930ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
762024-05-30T04:07:47.930ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
772024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
782024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
792024-05-30T04:07:47.931ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_6.vdev
802024-05-30T04:07:47.931ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
812024-05-30T04:07:47.931ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }), tx: ... } file = sled-storage/src/manager.rs:381
822024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
832024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
842024-05-30T04:07:47.931ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
852024-05-30T04:07:47.931ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_7.vdev
862024-05-30T04:07:47.931ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
872024-05-30T04:07:47.931ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }), tx: ... } file = sled-storage/src/manager.rs:381
882024-05-30T04:07:47.931ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
892024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
902024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
912024-05-30T04:07:47.931ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_8.vdev
922024-05-30T04:07:47.931ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
932024-05-30T04:07:47.931ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }), tx: ... } file = sled-storage/src/manager.rs:381
942024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
952024-05-30T04:07:47.931ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
962024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
972024-05-30T04:07:47.931ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:91
982024-05-30T04:07:47.931ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:93
992024-05-30T04:07:47.931ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
1002024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
1012024-05-30T04:07:47.931ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
1022024-05-30T04:07:47.931ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1032024-05-30T04:07:47.931ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:200
1042024-05-30T04:07:47.931ZINFOSledAgent (bootstore): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-05-30T04:07:47.931ZINFOSledAgent (bootstore): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-05-30T04:07:47.931ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-05-30T04:07:47.931ZINFOSledAgent (bootstore): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-05-30T04:07:47.931ZINFOSledAgent (bootstore): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-05-30T04:07:47.931ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-05-30T04:07:47.931ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:205
1112024-05-30T04:07:47.931ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:220
1122024-05-30T04:07:47.941ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:711
1132024-05-30T04:07:47.941ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 1264, tv_nsec: 30570189 } time_to_next_cleanup = 599.999926328s
1142024-05-30T04:07:47.941ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1152024-05-30T04:07:47.941ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3645
1162024-05-30T04:07:47.941ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-05-30T04:07:47.941ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-05-30T04:07:47.944ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-05-30T04:07:47.944ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-05-30T04:07:47.945ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV1 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV1 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], addresses: [ V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1004
1212024-05-30T04:07:47.945ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1222024-05-30T04:07:47.945ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1232024-05-30T04:07:47.945ZINFOSledAgent (RSS): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-05-30T04:07:47.945ZINFOSledAgent (RSS): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-05-30T04:07:47.945ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1064
1262024-05-30T04:07:47.945ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1272024-05-30T04:07:47.945ZINFOSledAgent (RSS): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-05-30T04:07:47.945ZINFOSledAgent (RSS): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-05-30T04:07:47.945ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1105
1302024-05-30T04:07:47.945ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:118
1312024-05-30T04:07:47.945ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:91
1322024-05-30T04:07:47.945ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1332024-05-30T04:07:47.945ZINFOSledAgent (RSS): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-05-30T04:07:47.945ZINFOSledAgent (RSS): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-05-30T04:07:47.945ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:138
1362024-05-30T04:07:47.945ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1138
1372024-05-30T04:07:47.945ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-05-30T04:07:47.945ZINFOSledAgent (bootstore): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-05-30T04:07:47.945ZINFOSledAgent (bootstore): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-05-30T04:07:47.946ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-05-30T04:07:47.946ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: dd908086-f372-434c-b66b-6e5770f567d8, rack_id: 4559c8be-8414-459c-b1cf-c6595c790cca, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-05-30T04:07:47.946ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:a02a::1]:33291
1432024-05-30T04:07:47.946ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: dd908086-f372-434c-b66b-6e5770f567d8, rack_id: 4559c8be-8414-459c-b1cf-c6595c790cca, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:360
1442024-05-30T04:07:47.946ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:375
1452024-05-30T04:07:47.946ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1462024-05-30T04:07:47.946ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1472024-05-30T04:07:47.946ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:381
1482024-05-30T04:07:47.946ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1492024-05-30T04:07:47.946ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:530
1502024-05-30T04:07:47.946ZINFOSledAgent (StorageManager): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1512024-05-30T04:07:47.946ZINFOSledAgent (StorageManager): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1522024-05-30T04:07:47.946ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:519
1532024-05-30T04:07:47.946ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:547
1542024-05-30T04:07:47.949ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3877
1552024-05-30T04:07:47.956ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
1562024-05-30T04:07:47.965ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
1572024-05-30T04:07:47.976ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
1582024-05-30T04:07:47.985ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
1592024-05-30T04:07:47.994ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
1602024-05-30T04:07:48.003ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
1612024-05-30T04:07:48.012ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
1622024-05-30T04:07:48.021ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
1632024-05-30T04:07:48.033ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
1642024-05-30T04:07:48.033ZINFOSledAgent (StorageManager): No disks to be automatically managed file = sled-storage/src/manager.rs:558
1652024-05-30T04:07:48.033ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:370
1662024-05-30T04:07:48.033ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
1672024-05-30T04:07:48.033ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:449 result = DisksManagementResult { status: [] }
1682024-05-30T04:07:48.033ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1692024-05-30T04:07:48.056ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:315
1702024-05-30T04:07:48.075ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1712024-05-30T04:07:48.085ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:353 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
1722024-05-30T04:07:48.085ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1732024-05-30T04:07:48.085ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:365 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
1742024-05-30T04:07:48.085ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1752024-05-30T04:07:48.088ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 145.333895ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d91df1e7-b013-4c8a-aa80-2400dc2b49ec", "content-length": "124", "date": "Thu, 30 May 2024 04:07:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d91df1e7-b013-4c8a-aa80-2400dc2b49ec" } file = clients/ddm-admin-client/src/lib.rs:178
1762024-05-30T04:07:48.098ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_75663f09-f516-40fc-852b-77e4a30319f5/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1772024-05-30T04:07:48.122ZINFOSledAgent: successfully created encrypted zvol: name="oxi_75663f09-f516-40fc-852b-77e4a30319f5/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1782024-05-30T04:07:48.122ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_75663f09-f516-40fc-852b-77e4a30319f5/swap" file = sled-agent/src/swap_device.rs:96
1792024-05-30T04:07:48.127ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:380 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
1802024-05-30T04:07:48.127ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1812024-05-30T04:07:48.127ZINFOSledAgent (BackingFs): Ensuring dataset oxi_75663f09-f516-40fc-852b-77e4a30319f5/backing/fmd file = sled-agent/src/backing_fs.rs:136
1822024-05-30T04:07:48.198ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1832024-05-30T04:07:48.234ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1842024-05-30T04:07:48.234ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 144.065623ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "00896ef7-f5c2-4989-8d69-1671b48f2860", "content-length": "124", "date": "Thu, 30 May 2024 04:07:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "00896ef7-f5c2-4989-8d69-1671b48f2860" } file = clients/ddm-admin-client/src/lib.rs:178
1852024-05-30T04:07:48.381ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1862024-05-30T04:07:48.381ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 204.58175ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "74c6382c-e269-418b-9854-19137bc134d0", "content-length": "124", "date": "Thu, 30 May 2024 04:07:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "74c6382c-e269-418b-9854-19137bc134d0" } file = clients/ddm-admin-client/src/lib.rs:178
1872024-05-30T04:07:48.417ZINFOSledAgent (BackingFs): Mounting oxi_75663f09-f516-40fc-852b-77e4a30319f5/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1882024-05-30T04:07:48.429ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1892024-05-30T04:07:48.429ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1902024-05-30T04:07:48.429ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1912024-05-30T04:07:48.429ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1922024-05-30T04:07:48.587ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1932024-05-30T04:07:48.587ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 169.32606ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "11e5fc82-eb25-45f8-8983-68b743edba89", "content-length": "124", "date": "Thu, 30 May 2024 04:07:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "11e5fc82-eb25-45f8-8983-68b743edba89" } file = clients/ddm-admin-client/src/lib.rs:178
1942024-05-30T04:07:48.759ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1952024-05-30T04:07:48.759ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 209.916819ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f5c3e036-c184-4d28-acf3-618e817a1a30", "content-length": "124", "date": "Thu, 30 May 2024 04:07:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f5c3e036-c184-4d28-acf3-618e817a1a30" } file = clients/ddm-admin-client/src/lib.rs:178
1962024-05-30T04:07:48.971ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1972024-05-30T04:07:48.971ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 324.662895ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3a87dbe6-7d3b-4c49-826e-fadebc5b8c37", "content-length": "124", "date": "Thu, 30 May 2024 04:07:48 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3a87dbe6-7d3b-4c49-826e-fadebc5b8c37" } file = clients/ddm-admin-client/src/lib.rs:178
1982024-05-30T04:07:49.153ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
1992024-05-30T04:07:49.153ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:39972 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2002024-05-30T04:07:49.153ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2012024-05-30T04:07:49.153ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:39972 file = oximeter/producer/src/lib.rs:276 interval = 30s producer_id = dd908086-f372-434c-b66b-6e5770f567d8 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2022024-05-30T04:07:49.159ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 361.926412ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2032024-05-30T04:07:49.200ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2042024-05-30T04:07:49.200ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2052024-05-30T04:07:49.200ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:993 underlay_address = fd00:1122:3344:101::1
2062024-05-30T04:07:49.200ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2072024-05-30T04:07:49.201ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:12345
2082024-05-30T04:07:49.201ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:430
2092024-05-30T04:07:49.201ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2102024-05-30T04:07:49.201ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2112024-05-30T04:07:49.202ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2122024-05-30T04:07:49.202ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2132024-05-30T04:07:49.202ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV1 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV1 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None }], addresses: [V4(Ipv4Net { addr: 10.151.2.174, width: 24 })], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2142024-05-30T04:07:49.202ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3645
2152024-05-30T04:07:49.202ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:a02a::1]:33291
2162024-05-30T04:07:49.396ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
2172024-05-30T04:07:49.396ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2182024-05-30T04:07:49.396ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2192024-05-30T04:07:49.396ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2202024-05-30T04:07:49.396ZINFOSledAgent (RSS): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2212024-05-30T04:07:49.396ZINFOSledAgent (RSS): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2222024-05-30T04:07:49.396ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2232024-05-30T04:07:49.396ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2242024-05-30T04:07:49.400ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2252024-05-30T04:07:49.428ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64552
2262024-05-30T04:07:49.428ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2272024-05-30T04:07:49.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64552 req_id = 08c74728-9637-4444-85f5-450f11cfed42 response_code = 200 uri = /inventory
2282024-05-30T04:07:49.457ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52111
2292024-05-30T04:07:49.457ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 58 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52111 req_id = e6c3b07d-bb59-444c-97fc-04bcf74e25d2 response_code = 200 uri = /sled-role
2302024-05-30T04:07:49.457ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2312024-05-30T04:07:49.457ZINFOSledAgent (RSS): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2322024-05-30T04:07:49.458ZINFOSledAgent (RSS): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2332024-05-30T04:07:49.458ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:815
2342024-05-30T04:07:49.489ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: 70e55cf7-953e-4643-9a59-9e2958d5bc87, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, pool_id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool) }, OmicronPhysicalDiskConfig { id: a5ae2404-80a9-4819-ba5e-6728732e3598, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, pool_id: 1793c88e-7478-4741-8254-860fae35146b (zpool) }, OmicronPhysicalDiskConfig { id: ef3821e0-ed58-4b96-afa2-47b3bd30eaef, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, pool_id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool) }, OmicronPhysicalDiskConfig { id: b9d5e503-3f68-4e2f-a22a-c4f8a87f5fdb, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, pool_id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool) }, OmicronPhysicalDiskConfig { id: 35cc8755-1c2b-423b-bedb-b6aeab45dbbb, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, pool_id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool) }, OmicronPhysicalDiskConfig { id: d10cffb8-7892-457e-a907-a90de19f4f0d, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, pool_id: a168fba6-15db-4743-814f-b76d3daf999f (zpool) }, OmicronPhysicalDiskConfig { id: 27df9c70-ad52-401d-a29d-6e10f7b3836a, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, pool_id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool) }, OmicronPhysicalDiskConfig { id: d40d83f5-e344-4e52-bad7-855a94aa96e3, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, pool_id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool) }, OmicronPhysicalDiskConfig { id: daa27901-b4fa-454a-be6d-f824b6019b4c, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, pool_id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool) }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:348 sled_address = [fd00:1122:3344:101::1]:12345
2352024-05-30T04:07:49.489ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64395
2362024-05-30T04:07:49.490ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 70e55cf7-953e-4643-9a59-9e2958d5bc87, pool_id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: a5ae2404-80a9-4819-ba5e-6728732e3598, pool_id: 1793c88e-7478-4741-8254-860fae35146b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: ef3821e0-ed58-4b96-afa2-47b3bd30eaef, pool_id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: b9d5e503-3f68-4e2f-a22a-c4f8a87f5fdb, pool_id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 35cc8755-1c2b-423b-bedb-b6aeab45dbbb, pool_id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: d10cffb8-7892-457e-a907-a90de19f4f0d, pool_id: a168fba6-15db-4743-814f-b76d3daf999f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 27df9c70-ad52-401d-a29d-6e10f7b3836a, pool_id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: d40d83f5-e344-4e52-bad7-855a94aa96e3, pool_id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: daa27901-b4fa-454a-be6d-f824b6019b4c, pool_id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:381
2372024-05-30T04:07:49.490ZINFOSledAgent (StorageManager): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2382024-05-30T04:07:49.490ZINFOSledAgent (StorageManager): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2392024-05-30T04:07:49.490ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:681 request = omicron_physical_disks_ensure
2402024-05-30T04:07:49.490ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
2412024-05-30T04:07:49.490ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:379
2422024-05-30T04:07:49.490ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2432024-05-30T04:07:49.490ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2442024-05-30T04:07:49.606ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 352.708911ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2452024-05-30T04:07:49.612ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:304
2462024-05-30T04:07:49.612ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool)
2472024-05-30T04:07:50.011ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 523.904099ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2482024-05-30T04:07:50.279ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2492024-05-30T04:07:50.684ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.93510184s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2502024-05-30T04:07:50.693ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }
2512024-05-30T04:07:50.889ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:332
2522024-05-30T04:07:50.889ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:334
2532024-05-30T04:07:50.889ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:338
2542024-05-30T04:07:50.889ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:340
2552024-05-30T04:07:50.889ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2562024-05-30T04:07:50.889ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2572024-05-30T04:07:50.889ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2582024-05-30T04:07:51.305ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_switch
2592024-05-30T04:07:51.740ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2602024-05-30T04:07:51.740ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2612024-05-30T04:07:51.960ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2622024-05-30T04:07:52.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 47.730227ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2632024-05-30T04:07:52.173ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_3022fa0e-6566-4854-a1a9-1e0c5908d5b0/crypt/zone file = sled-storage/src/dataset.rs:395
2642024-05-30T04:07:52.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 106.656691ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2652024-05-30T04:07:52.774ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 5.138147591s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2662024-05-30T04:07:52.774ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2672024-05-30T04:07:53.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 158.827502ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2682024-05-30T04:07:53.438ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2692024-05-30T04:07:54.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 493.231977ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2702024-05-30T04:07:54.037ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2712024-05-30T04:07:55.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.070324886s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2722024-05-30T04:07:55.104ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2732024-05-30T04:07:55.114ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_3022fa0e-6566-4854-a1a9-1e0c5908d5b0 file = sled-storage/src/dataset.rs:480
2742024-05-30T04:07:55.309ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:477
2752024-05-30T04:07:55.309ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:394
2762024-05-30T04:07:55.309ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:379
2772024-05-30T04:07:55.309ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2782024-05-30T04:07:55.309ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2792024-05-30T04:07:55.509ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:304
2802024-05-30T04:07:55.509ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 1793c88e-7478-4741-8254-860fae35146b (zpool)
2812024-05-30T04:07:56.570ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }
2822024-05-30T04:07:56.798ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:332
2832024-05-30T04:07:56.798ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:334
2842024-05-30T04:07:56.798ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:338
2852024-05-30T04:07:56.798ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:340
2862024-05-30T04:07:56.798ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2872024-05-30T04:07:56.798ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2882024-05-30T04:07:56.798ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2892024-05-30T04:07:57.669ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2902024-05-30T04:07:57.901ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2912024-05-30T04:07:58.121ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2922024-05-30T04:07:58.340ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 11.81069839s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2932024-05-30T04:07:58.578ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1793c88e-7478-4741-8254-860fae35146b/crypt/zone file = sled-storage/src/dataset.rs:395
2942024-05-30T04:07:59.230ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2952024-05-30T04:07:59.230ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2962024-05-30T04:07:59.432ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
2972024-05-30T04:07:59.540ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1793c88e-7478-4741-8254-860fae35146b file = sled-storage/src/dataset.rs:480
2982024-05-30T04:07:59.548ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:477
2992024-05-30T04:07:59.548ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:394
3002024-05-30T04:07:59.548ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:379
3012024-05-30T04:07:59.548ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3022024-05-30T04:07:59.548ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3032024-05-30T04:07:59.558ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:304
3042024-05-30T04:07:59.558ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = f61d361f-e4c0-48df-9191-ff705041ee76 (zpool)
3052024-05-30T04:07:59.653ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }
3062024-05-30T04:07:59.661ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:332
3072024-05-30T04:07:59.661ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3082024-05-30T04:07:59.661ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3092024-05-30T04:07:59.661ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3102024-05-30T04:07:59.661ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3112024-05-30T04:07:59.661ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3122024-05-30T04:07:59.661ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3132024-05-30T04:07:59.695ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3142024-05-30T04:07:59.704ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f61d361f-e4c0-48df-9191-ff705041ee76/crypt/zone file = sled-storage/src/dataset.rs:395
3152024-05-30T04:07:59.826ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f61d361f-e4c0-48df-9191-ff705041ee76 file = sled-storage/src/dataset.rs:480
3162024-05-30T04:07:59.834ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:477
3172024-05-30T04:07:59.834ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:394
3182024-05-30T04:07:59.834ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:379
3192024-05-30T04:07:59.834ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3202024-05-30T04:07:59.834ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3212024-05-30T04:07:59.843ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:304
3222024-05-30T04:07:59.843ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool)
3232024-05-30T04:07:59.935ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }
3242024-05-30T04:07:59.943ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:332
3252024-05-30T04:07:59.943ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:334
3262024-05-30T04:07:59.943ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:338
3272024-05-30T04:07:59.943ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:340
3282024-05-30T04:07:59.943ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3292024-05-30T04:07:59.943ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3302024-05-30T04:07:59.943ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3312024-05-30T04:07:59.976ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3322024-05-30T04:07:59.983ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7/crypt/zone file = sled-storage/src/dataset.rs:395
3332024-05-30T04:08:00.106ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 file = sled-storage/src/dataset.rs:480
3342024-05-30T04:08:00.113ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:477
3352024-05-30T04:08:00.113ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:394
3362024-05-30T04:08:00.113ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:379
3372024-05-30T04:08:00.113ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3382024-05-30T04:08:00.113ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3392024-05-30T04:08:00.123ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:304
3402024-05-30T04:08:00.123ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool)
3412024-05-30T04:08:00.219ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }
3422024-05-30T04:08:00.227ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:332
3432024-05-30T04:08:00.227ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:334
3442024-05-30T04:08:00.227ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:338
3452024-05-30T04:08:00.227ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:340
3462024-05-30T04:08:00.227ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3472024-05-30T04:08:00.227ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3482024-05-30T04:08:00.227ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3492024-05-30T04:08:00.232ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
3502024-05-30T04:08:00.262ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3512024-05-30T04:08:00.270ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_5d12239f-a80e-4f75-83d4-ebef24b6fea1/crypt/zone file = sled-storage/src/dataset.rs:395
3522024-05-30T04:08:00.392ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_5d12239f-a80e-4f75-83d4-ebef24b6fea1 file = sled-storage/src/dataset.rs:480
3532024-05-30T04:08:00.400ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:477
3542024-05-30T04:08:00.400ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:394
3552024-05-30T04:08:00.400ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:379
3562024-05-30T04:08:00.400ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3572024-05-30T04:08:00.400ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3582024-05-30T04:08:00.409ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:304
3592024-05-30T04:08:00.409ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = a168fba6-15db-4743-814f-b76d3daf999f (zpool)
3602024-05-30T04:08:00.501ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }
3612024-05-30T04:08:00.509ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:332
3622024-05-30T04:08:00.509ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:334
3632024-05-30T04:08:00.509ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:338
3642024-05-30T04:08:00.509ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:340
3652024-05-30T04:08:00.509ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3662024-05-30T04:08:00.509ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3672024-05-30T04:08:00.509ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3682024-05-30T04:08:00.545ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3692024-05-30T04:08:00.554ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a168fba6-15db-4743-814f-b76d3daf999f/crypt/zone file = sled-storage/src/dataset.rs:395
3702024-05-30T04:08:00.676ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a168fba6-15db-4743-814f-b76d3daf999f file = sled-storage/src/dataset.rs:480
3712024-05-30T04:08:00.684ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:477
3722024-05-30T04:08:00.684ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:394
3732024-05-30T04:08:00.684ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:379
3742024-05-30T04:08:00.684ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3752024-05-30T04:08:00.684ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3762024-05-30T04:08:00.693ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:304
3772024-05-30T04:08:00.693ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool)
3782024-05-30T04:08:00.788ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }
3792024-05-30T04:08:00.796ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:332
3802024-05-30T04:08:00.796ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:334
3812024-05-30T04:08:00.796ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:338
3822024-05-30T04:08:00.796ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:340
3832024-05-30T04:08:00.796ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3842024-05-30T04:08:00.796ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3852024-05-30T04:08:00.796ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3862024-05-30T04:08:00.800ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
3872024-05-30T04:08:00.834ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3882024-05-30T04:08:00.843ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crypt/zone file = sled-storage/src/dataset.rs:395
3892024-05-30T04:08:00.918ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2331
3902024-05-30T04:08:00.968ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 file = sled-storage/src/dataset.rs:480
3912024-05-30T04:08:00.975ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:477
3922024-05-30T04:08:00.975ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:394
3932024-05-30T04:08:00.975ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:379
3942024-05-30T04:08:00.975ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3952024-05-30T04:08:00.975ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3962024-05-30T04:08:00.984ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:304
3972024-05-30T04:08:00.985ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool)
3982024-05-30T04:08:01.079ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }
3992024-05-30T04:08:01.086ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:332
4002024-05-30T04:08:01.086ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:334
4012024-05-30T04:08:01.086ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:338
4022024-05-30T04:08:01.086ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:340
4032024-05-30T04:08:01.086ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4042024-05-30T04:08:01.087ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4052024-05-30T04:08:01.087ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4062024-05-30T04:08:01.121ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4072024-05-30T04:08:01.130ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_dc70dd36-578e-4d7e-9ca2-11c36573efc3/crypt/zone file = sled-storage/src/dataset.rs:395
4082024-05-30T04:08:01.234ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
4092024-05-30T04:08:01.254ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_dc70dd36-578e-4d7e-9ca2-11c36573efc3 file = sled-storage/src/dataset.rs:480
4102024-05-30T04:08:01.262ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:477
4112024-05-30T04:08:01.262ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:394
4122024-05-30T04:08:01.262ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:379
4132024-05-30T04:08:01.262ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
4142024-05-30T04:08:01.262ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4152024-05-30T04:08:01.271ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:304
4162024-05-30T04:08:01.271ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool)
4172024-05-30T04:08:01.367ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }
4182024-05-30T04:08:01.374ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:332
4192024-05-30T04:08:01.374ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:334
4202024-05-30T04:08:01.374ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:338
4212024-05-30T04:08:01.374ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:340
4222024-05-30T04:08:01.374ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4232024-05-30T04:08:01.374ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4242024-05-30T04:08:01.374ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4252024-05-30T04:08:01.410ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4262024-05-30T04:08:01.418ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8f547a7a-6177-4ca0-82ce-6646cf05d8a6/crypt/zone file = sled-storage/src/dataset.rs:395
4272024-05-30T04:08:01.542ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8f547a7a-6177-4ca0-82ce-6646cf05d8a6 file = sled-storage/src/dataset.rs:480
4282024-05-30T04:08:01.550ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:477
4292024-05-30T04:08:01.550ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:394
4302024-05-30T04:08:01.550ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4312024-05-30T04:08:01.550ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 75663f09-f516-40fc-852b-77e4a30319f5 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 804b9788-bd41-4d87-a91b-e568478fa644 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }, zpool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }, zpool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }, zpool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }, zpool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }, zpool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }, zpool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }, zpool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }, zpool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }, zpool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
4322024-05-30T04:08:01.550ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
4332024-05-30T04:08:01.550ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
4342024-05-30T04:08:01.550ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4352024-05-30T04:08:01.551ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 12061150 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64395 req_id = 6b486cd9-72dd-4ba2-a5a5-7acbb35f629c response_code = 200 uri = /omicron-physical-disks
4362024-05-30T04:08:01.581ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
4372024-05-30T04:08:01.581ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53600
4382024-05-30T04:08:01.581ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4392024-05-30T04:08:01.581ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4402024-05-30T04:08:01.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4412024-05-30T04:08:01.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
4422024-05-30T04:08:01.681ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4432024-05-30T04:08:01.681ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4442024-05-30T04:08:01.732ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4452024-05-30T04:08:01.732ZINFOSledAgent (ServiceManager): No ledger in /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:188
4462024-05-30T04:08:01.732ZINFOSledAgent (ServiceManager): No ledger in /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:188
4472024-05-30T04:08:01.732ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4482024-05-30T04:08:01.776ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4492024-05-30T04:08:01.776ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4502024-05-30T04:08:01.776ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4512024-05-30T04:08:01.826ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063 file = illumos-utils/src/zone.rs:315
4522024-05-30T04:08:01.829ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982 file = illumos-utils/src/zone.rs:315
4532024-05-30T04:08:01.833ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd file = illumos-utils/src/zone.rs:315
4542024-05-30T04:08:01.845ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063 file = illumos-utils/src/zone.rs:354
4552024-05-30T04:08:01.850ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982 file = illumos-utils/src/zone.rs:354
4562024-05-30T04:08:01.854ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd file = illumos-utils/src/zone.rs:354
4572024-05-30T04:08:02.236ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
4582024-05-30T04:08:02.801ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/dd908086-f372-434c-b66b-6e5770f567d8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
4592024-05-30T04:08:03.364ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4602024-05-30T04:08:03.364ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
4612024-05-30T04:08:03.844ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4622024-05-30T04:08:04.322ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
4632024-05-30T04:08:04.322ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4642024-05-30T04:08:04.322ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
4652024-05-30T04:08:04.326ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
4662024-05-30T04:08:04.326ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4672024-05-30T04:08:04.330ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
4682024-05-30T04:08:04.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd") failed: Property not found. retry in 68.889482ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
4692024-05-30T04:08:04.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 40.069723ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4702024-05-30T04:08:04.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982") failed: Property not found. retry in 25.333895ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
4712024-05-30T04:08:04.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982") failed: Property not found. retry in 56.673419ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
4722024-05-30T04:08:04.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 52.554987ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4732024-05-30T04:08:04.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd") failed: Property not found. retry in 113.961931ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
4742024-05-30T04:08:04.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 170.528916ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4752024-05-30T04:08:04.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982") failed: Property not found. retry in 278.59887ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
4762024-05-30T04:08:04.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd") failed: Property not found. retry in 126.61523ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
4772024-05-30T04:08:04.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 234.11341ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4782024-05-30T04:08:05.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982") failed: Property not found. retry in 567.830602ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
4792024-05-30T04:08:05.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd") failed: Property not found. retry in 307.225684ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
4802024-05-30T04:08:05.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 447.532451ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4812024-05-30T04:08:05.389ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd") failed: Property not found. retry in 825.876859ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
4822024-05-30T04:08:05.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982") failed: Property not found. retry in 1.182060878s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
4832024-05-30T04:08:05.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 770.423937ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4842024-05-30T04:08:06.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd") failed: Property not found. retry in 1.354237491s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
4852024-05-30T04:08:06.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 1.172663259s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4862024-05-30T04:08:06.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982") failed: Property not found. retry in 592.901854ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
4872024-05-30T04:08:07.004ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:a02a::2 exists in switch zone file = sled-agent/src/services.rs:2348
4882024-05-30T04:08:07.004ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:626 zone = oxz_switch
4892024-05-30T04:08:07.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982") failed: Property not found. retry in 1.291282407s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
4902024-05-30T04:08:07.557ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fe8e:5929 file = sled-agent/src/services.rs:2355
4912024-05-30T04:08:07.563ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2416
4922024-05-30T04:08:07.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd") failed: Property not found. retry in 1.023072949s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
4932024-05-30T04:08:07.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 515.717398ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4942024-05-30T04:08:07.683ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2601
4952024-05-30T04:08:07.920ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2765
4962024-05-30T04:08:08.106ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2480
4972024-05-30T04:08:08.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 527.851008ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
4982024-05-30T04:08:08.396ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2515
4992024-05-30T04:08:08.594ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2815
5002024-05-30T04:08:08.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd") failed: Property not found. retry in 1.283671878s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
5012024-05-30T04:08:08.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063") failed: Property not found. retry in 1.342999536s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
5022024-05-30T04:08:08.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982") failed: Property not found. retry in 1.30904213s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
5032024-05-30T04:08:08.768ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2842
5042024-05-30T04:08:08.987ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2709
5052024-05-30T04:08:09.162ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2509
5062024-05-30T04:08:09.173ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3901 new = [fd00:1122:3344:101::2, ::1] old = [::1]
5072024-05-30T04:08:09.173ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3917
5082024-05-30T04:08:09.173ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:609 zone = oxz_switch
5092024-05-30T04:08:09.723ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3925
5102024-05-30T04:08:09.729ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3981
5112024-05-30T04:08:09.876ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4043
5122024-05-30T04:08:10.000ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4024 rack_subnet = fd00:1122:3344::
5132024-05-30T04:08:10.028ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4071
5142024-05-30T04:08:10.032ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_internal_dns_a5f19f92-dd9d-4d5d-b88b-df21ddc95063
5152024-05-30T04:08:10.032ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_internal_dns_5269a058-580b-4ad2-a686-a68f8b8e0982
5162024-05-30T04:08:10.032ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_internal_dns_448a7523-abaa-4f37-8ea0-88fd1a15dcfd
5172024-05-30T04:08:10.032ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5182024-05-30T04:08:10.032ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5192024-05-30T04:08:10.032ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8451175 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53600 req_id = de22cb3f-3ee5-4521-a2be-6e336f16c17e response_code = 204 uri = /omicron-zones
5202024-05-30T04:08:10.034ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5212024-05-30T04:08:10.163ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4098
5222024-05-30T04:08:10.308ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:365
5232024-05-30T04:08:10.339ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:413
5242024-05-30T04:08:10.370ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:651
5252024-05-30T04:08:10.370ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:689
5262024-05-30T04:08:10.371ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:692 uptime = 3
5272024-05-30T04:08:10.371ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:435
5282024-05-30T04:08:10.449ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
5292024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5302024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5312024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5322024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5332024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5342024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5352024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5362024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5372024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5382024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5392024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5402024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5412024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5422024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5432024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5442024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5452024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5462024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5472024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5482024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5492024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5502024-05-30T04:08:10.449ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5512024-05-30T04:08:10.639ZWARNSledAgent (RSS): failed to write DNS configuration (will retry in 89.397296ms) dns_config_addr = [fd00:1122:3344:1::1]:5353 error_message = Communication Error: error sending request for url (http://[fd00:1122:3344:1::1]:5353/config): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:569
5522024-05-30T04:08:10.748ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5532024-05-30T04:08:10.795ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5542024-05-30T04:08:10.844ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:585
5552024-05-30T04:08:10.844ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
5562024-05-30T04:08:10.875ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
5572024-05-30T04:08:10.877ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
5582024-05-30T04:08:10.877ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
5592024-05-30T04:08:10.907ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
5602024-05-30T04:08:10.907ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42994
5612024-05-30T04:08:10.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5622024-05-30T04:08:10.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5632024-05-30T04:08:10.929ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5642024-05-30T04:08:10.929ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
5652024-05-30T04:08:10.948ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5662024-05-30T04:08:10.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5672024-05-30T04:08:10.966ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5682024-05-30T04:08:10.966ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
5692024-05-30T04:08:10.966ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
5702024-05-30T04:08:10.966ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5712024-05-30T04:08:10.978ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
5722024-05-30T04:08:11.011ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
5732024-05-30T04:08:11.012ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
5742024-05-30T04:08:11.012ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
5752024-05-30T04:08:11.056ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 169, 65]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
5762024-05-30T04:08:11.056ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = ntp
5772024-05-30T04:08:11.057ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
5782024-05-30T04:08:11.057ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
5792024-05-30T04:08:11.057ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
5802024-05-30T04:08:11.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5812024-05-30T04:08:11.083ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d file = illumos-utils/src/zone.rs:315
5822024-05-30T04:08:11.101ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d file = illumos-utils/src/zone.rs:354
5832024-05-30T04:08:11.718ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5842024-05-30T04:08:11.718ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5852024-05-30T04:08:11.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 57.32953ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5862024-05-30T04:08:11.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 110.890964ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5872024-05-30T04:08:12.064ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 121.337388ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5882024-05-30T04:08:12.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 407.872749ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5892024-05-30T04:08:12.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 642.716246ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5902024-05-30T04:08:13.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 862.086763ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5912024-05-30T04:08:13.679ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36630
5922024-05-30T04:08:13.679ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 65 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36630 req_id = 8d3c54ff-aec4-4995-b70f-8e3cf20722e4 response_code = 200 uri = /baseboard
5932024-05-30T04:08:14.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 506.087932ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5942024-05-30T04:08:14.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 706.307785ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5952024-05-30T04:08:15.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 648.616462ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5962024-05-30T04:08:16.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d") failed: Property not found. retry in 643.845596ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5972024-05-30T04:08:16.691ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d
5982024-05-30T04:08:16.691ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5992024-05-30T04:08:16.691ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6002024-05-30T04:08:16.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 5784648 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:42994 req_id = 961d094e-cc5f-450c-8279-ff20ed752686 response_code = 204 uri = /omicron-zones
6012024-05-30T04:08:16.693ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:626
6022024-05-30T04:08:16.722ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6032024-05-30T04:08:16.722ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42525
6042024-05-30T04:08:16.733ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3631
6052024-05-30T04:08:16.733ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 10706 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42525 req_id = 00c1815d-2920-4e2e-9eb6-096243c735cd response_code = 500 uri = /timesync
6062024-05-30T04:08:16.736ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6072024-05-30T04:08:16.819ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6082024-05-30T04:08:16.819ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59146
6092024-05-30T04:08:18.828ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3631
6102024-05-30T04:08:18.828ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 2008777 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59146 req_id = 8689eb65-ad7c-4f56-8277-9283e14413ab response_code = 500 uri = /timesync
6112024-05-30T04:08:18.830ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6122024-05-30T04:08:18.976ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6132024-05-30T04:08:18.976ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51222
6142024-05-30T04:08:18.985ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3631
6152024-05-30T04:08:18.985ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 8573 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51222 req_id = eb4e4c15-bc27-4bbc-a29e-3948633b284c response_code = 500 uri = /timesync
6162024-05-30T04:08:18.987ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6172024-05-30T04:08:19.099ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6182024-05-30T04:08:19.100ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61120
6192024-05-30T04:08:23.107ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_37aef7a7-53c8-4b7f-aab3-5292349f3f5d': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3631
6202024-05-30T04:08:23.107ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 4007689 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61120 req_id = 5d138095-8a81-4a3f-8353-c1691c55d625 response_code = 500 uri = /timesync
6212024-05-30T04:08:23.110ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6222024-05-30T04:08:23.255ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6232024-05-30T04:08:23.255ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65195
6242024-05-30T04:08:23.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 9081 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65195 req_id = 46fa1e6d-a9c0-4d92-8bce-9e9e1a9fa614 response_code = 200 uri = /timesync
6252024-05-30T04:08:23.267ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6262024-05-30T04:08:23.267ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6272024-05-30T04:08:23.590ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6282024-05-30T04:08:23.590ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61889
6292024-05-30T04:08:23.598ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7940 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61889 req_id = 1442ad5d-77be-447f-8f53-8bcbd9de3313 response_code = 200 uri = /timesync
6302024-05-30T04:08:23.600ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6312024-05-30T04:08:23.600ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6322024-05-30T04:08:23.953ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6332024-05-30T04:08:23.953ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39470
6342024-05-30T04:08:23.962ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8490 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39470 req_id = 75f083e7-3ae1-49a5-a5f6-21c7c77139e5 response_code = 200 uri = /timesync
6352024-05-30T04:08:23.963ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6362024-05-30T04:08:23.963ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6372024-05-30T04:08:24.401ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6382024-05-30T04:08:24.401ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54670
6392024-05-30T04:08:24.409ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7781 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54670 req_id = b42be9f4-2934-47a8-a3ea-f030bcf6f22b response_code = 200 uri = /timesync
6402024-05-30T04:08:24.412ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6412024-05-30T04:08:24.412ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6422024-05-30T04:08:24.866ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6432024-05-30T04:08:24.866ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49939
6442024-05-30T04:08:24.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8589 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49939 req_id = a31d4ca4-7c92-4f1d-9b33-0d14a5a87615 response_code = 200 uri = /timesync
6452024-05-30T04:08:24.878ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6462024-05-30T04:08:24.878ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6472024-05-30T04:08:25.435ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6482024-05-30T04:08:25.435ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59407
6492024-05-30T04:08:25.443ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8013 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59407 req_id = 75a17a89-4ca3-4ed6-a5d7-8f1fe92b7120 response_code = 200 uri = /timesync
6502024-05-30T04:08:25.445ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6512024-05-30T04:08:25.445ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6522024-05-30T04:08:26.220ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6532024-05-30T04:08:26.220ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48809
6542024-05-30T04:08:26.229ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8521 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48809 req_id = 8e090a5b-2e2c-49b5-8b7b-1085a7b2d9ed response_code = 200 uri = /timesync
6552024-05-30T04:08:26.231ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6562024-05-30T04:08:26.231ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6572024-05-30T04:08:26.632ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6582024-05-30T04:08:26.632ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55273
6592024-05-30T04:08:26.641ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8617 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55273 req_id = f5836997-8442-4fcb-88cc-4ad555cd62c7 response_code = 200 uri = /timesync
6602024-05-30T04:08:26.644ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6612024-05-30T04:08:26.644ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6622024-05-30T04:08:27.504ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6632024-05-30T04:08:27.504ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58537
6642024-05-30T04:08:27.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7886 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58537 req_id = 6eb85e3e-a178-4ae6-bf48-eff64ad174da response_code = 200 uri = /timesync
6652024-05-30T04:08:27.514ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6662024-05-30T04:08:27.514ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6672024-05-30T04:08:28.278ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6682024-05-30T04:08:28.278ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48210
6692024-05-30T04:08:28.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 20621 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48210 req_id = 31cec257-e59d-4b6f-a100-840859be7c6a response_code = 200 uri = /timesync
6702024-05-30T04:08:28.300ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2310930948, ip_addr: 137.190.2.4, stratum: 3, ref_time: 1717042106.539167, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6712024-05-30T04:08:28.300ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1217
6722024-05-30T04:08:28.329ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
6732024-05-30T04:08:28.329ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64117
6742024-05-30T04:08:28.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6752024-05-30T04:08:28.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6762024-05-30T04:08:28.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6772024-05-30T04:08:28.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
6782024-05-30T04:08:28.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6792024-05-30T04:08:28.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6802024-05-30T04:08:28.384ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6812024-05-30T04:08:28.384ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6822024-05-30T04:08:28.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6832024-05-30T04:08:28.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6842024-05-30T04:08:28.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6852024-05-30T04:08:28.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6862024-05-30T04:08:28.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6872024-05-30T04:08:28.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6882024-05-30T04:08:28.583ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6892024-05-30T04:08:28.583ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6902024-05-30T04:08:28.634ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6912024-05-30T04:08:28.634ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
6922024-05-30T04:08:28.635ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
6932024-05-30T04:08:28.635ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6942024-05-30T04:08:28.717ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6952024-05-30T04:08:28.717ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6962024-05-30T04:08:28.717ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6972024-05-30T04:08:28.717ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6982024-05-30T04:08:28.717ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6992024-05-30T04:08:28.805ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b file = illumos-utils/src/zone.rs:315
7002024-05-30T04:08:28.809ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035 file = illumos-utils/src/zone.rs:315
7012024-05-30T04:08:28.813ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690 file = illumos-utils/src/zone.rs:315
7022024-05-30T04:08:28.818ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456 file = illumos-utils/src/zone.rs:315
7032024-05-30T04:08:28.822ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0 file = illumos-utils/src/zone.rs:315
7042024-05-30T04:08:28.826ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b file = illumos-utils/src/zone.rs:354
7052024-05-30T04:08:28.830ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035 file = illumos-utils/src/zone.rs:354
7062024-05-30T04:08:28.835ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690 file = illumos-utils/src/zone.rs:354
7072024-05-30T04:08:28.839ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456 file = illumos-utils/src/zone.rs:354
7082024-05-30T04:08:28.844ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0 file = illumos-utils/src/zone.rs:354
7092024-05-30T04:08:30.907ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 204.901214ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
7102024-05-30T04:08:32.537ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::5]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::5]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::5]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7112024-05-30T04:08:32.537ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::4"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::4]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::4]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::4]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7122024-05-30T04:08:32.537ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7132024-05-30T04:08:32.543ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7142024-05-30T04:08:32.577ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::6]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::6]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::6]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7152024-05-30T04:08:32.578ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::3"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::3]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::3]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::3]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7162024-05-30T04:08:32.578ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7172024-05-30T04:08:32.583ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7182024-05-30T04:08:32.591ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::7]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::7]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::7]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7192024-05-30T04:08:32.592ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7202024-05-30T04:08:32.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 50.030945ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7212024-05-30T04:08:32.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 64.481584ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7222024-05-30T04:08:32.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 65.707614ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7232024-05-30T04:08:32.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 30.066721ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7242024-05-30T04:08:32.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 68.180386ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7252024-05-30T04:08:32.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 123.864902ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7262024-05-30T04:08:33.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 144.564436ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7272024-05-30T04:08:33.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 140.551119ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7282024-05-30T04:08:34.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 113.528084ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7292024-05-30T04:08:34.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 150.288707ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7302024-05-30T04:08:34.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 172.844279ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7312024-05-30T04:08:34.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 98.192434ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7322024-05-30T04:08:34.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 236.854594ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7332024-05-30T04:08:34.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 155.010151ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7342024-05-30T04:08:34.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 320.090086ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7352024-05-30T04:08:35.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 276.448763ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7362024-05-30T04:08:35.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 466.477567ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7372024-05-30T04:08:35.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 230.382939ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7382024-05-30T04:08:35.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 599.918237ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7392024-05-30T04:08:35.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 422.714506ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7402024-05-30T04:08:35.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 662.28657ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7412024-05-30T04:08:35.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 253.577612ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7422024-05-30T04:08:35.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 442.006455ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7432024-05-30T04:08:35.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 662.35189ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7442024-05-30T04:08:36.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 990.641127ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7452024-05-30T04:08:36.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 789.11932ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7462024-05-30T04:08:36.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 1.085255804s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7472024-05-30T04:08:36.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 1.030959505s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7482024-05-30T04:08:36.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 623.685334ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7492024-05-30T04:08:36.972ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 613.07901ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7502024-05-30T04:08:37.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 1.470519123s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7512024-05-30T04:08:37.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 1.344691308s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7522024-05-30T04:08:37.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 563.743037ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7532024-05-30T04:08:37.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 1.156537442s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7542024-05-30T04:08:37.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 883.827054ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7552024-05-30T04:08:37.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 773.660105ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7562024-05-30T04:08:38.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 1.130102454s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7572024-05-30T04:08:38.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 1.348001555s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7582024-05-30T04:08:38.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 1.479819742s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7592024-05-30T04:08:38.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 980.511603ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7602024-05-30T04:08:38.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 656.892098ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7612024-05-30T04:08:39.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 1.071025564s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7622024-05-30T04:08:39.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 808.285516ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7632024-05-30T04:08:39.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 1.367039343s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7642024-05-30T04:08:39.937ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 1.17396571s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7652024-05-30T04:08:40.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 1.391676271s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7662024-05-30T04:08:40.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 1.471077383s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7672024-05-30T04:08:40.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 686.023985ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7682024-05-30T04:08:41.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 733.937073ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7692024-05-30T04:08:41.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 889.666371ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7702024-05-30T04:08:41.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 1.292802015s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7712024-05-30T04:08:41.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035") failed: Property not found. retry in 1.058621215s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7722024-05-30T04:08:41.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 1.338391041s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7732024-05-30T04:08:41.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0") failed: Property not found. retry in 931.392505ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7742024-05-30T04:08:42.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690") failed: Property not found. retry in 581.282284ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7752024-05-30T04:08:42.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456") failed: Property not found. retry in 863.306476ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7762024-05-30T04:08:43.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 634.08709ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7772024-05-30T04:08:43.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 581.150561ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7782024-05-30T04:08:44.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 736.439139ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7792024-05-30T04:08:44.450ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:43177
7802024-05-30T04:08:44.450ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 92 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:43177 req_id = 67820c20-4e1f-4885-8da7-2537e4b2c8ad response_code = 200 uri = /baseboard
7812024-05-30T04:08:45.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 625.592505ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7822024-05-30T04:08:46.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 747.393684ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7832024-05-30T04:08:46.265ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 675.280773ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
7842024-05-30T04:08:46.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 1.195558181s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7852024-05-30T04:08:48.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 676.809094ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7862024-05-30T04:08:48.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 1.227757339s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7872024-05-30T04:08:50.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 1.346392462s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7882024-05-30T04:08:51.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b") failed: Property not found. retry in 1.22305523s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7892024-05-30T04:08:52.677ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b
7902024-05-30T04:08:52.677ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_cockroachdb_72c3976d-68e9-463a-be70-3de38abfe035
7912024-05-30T04:08:52.677ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_cockroachdb_9112ad3c-6c15-4601-bf24-48148ee77690
7922024-05-30T04:08:52.677ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_cockroachdb_a84ccf3e-ca81-499f-b526-5c900c243456
7932024-05-30T04:08:52.677ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_cockroachdb_fef621b3-c922-4648-a07b-e45ce7dbe3e0
7942024-05-30T04:08:52.677ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7952024-05-30T04:08:52.678ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7962024-05-30T04:08:52.721ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 23602692 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64117 req_id = 5f36bab8-1467-41ed-bd47-85c535d1b2ce response_code = 204 uri = /omicron-zones
7972024-05-30T04:08:52.754ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51068
7982024-05-30T04:08:52.789ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 34856 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = 582d0c1d-1cd3-4d2b-a1b3-248378dcfc03 response_code = 500 uri = /cockroachdb
7992024-05-30T04:08:52.789ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "582d0c1d-1cd3-4d2b-a1b3-248378dcfc03", "content-length": "124", "date": "Thu, 30 May 2024 04:08:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "582d0c1d-1cd3-4d2b-a1b3-248378dcfc03" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 92.830849ms
8002024-05-30T04:08:52.911ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 27357 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = 8d743e38-cbb4-4d8b-ac86-560c7ce7b65b response_code = 500 uri = /cockroachdb
8012024-05-30T04:08:52.911ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8d743e38-cbb4-4d8b-ac86-560c7ce7b65b", "content-length": "124", "date": "Thu, 30 May 2024 04:08:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8d743e38-cbb4-4d8b-ac86-560c7ce7b65b" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 104.159373ms
8022024-05-30T04:08:53.042ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 26056 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = 6eb262a5-068f-4841-a1e9-2267fcb2d117 response_code = 500 uri = /cockroachdb
8032024-05-30T04:08:53.042ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6eb262a5-068f-4841-a1e9-2267fcb2d117", "content-length": "124", "date": "Thu, 30 May 2024 04:08:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6eb262a5-068f-4841-a1e9-2267fcb2d117" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 72.082994ms
8042024-05-30T04:08:53.141ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 25970 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = da4a31f4-ef96-4fce-a7d2-e8e8c0d25ab5 response_code = 500 uri = /cockroachdb
8052024-05-30T04:08:53.142ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "da4a31f4-ef96-4fce-a7d2-e8e8c0d25ab5", "content-length": "124", "date": "Thu, 30 May 2024 04:08:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "da4a31f4-ef96-4fce-a7d2-e8e8c0d25ab5" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 110.414703ms
8062024-05-30T04:08:53.279ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 25944 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = ba3aaab6-d87c-4a41-8964-9ffc87f94b4d response_code = 500 uri = /cockroachdb
8072024-05-30T04:08:53.279ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ba3aaab6-d87c-4a41-8964-9ffc87f94b4d", "content-length": "124", "date": "Thu, 30 May 2024 04:08:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ba3aaab6-d87c-4a41-8964-9ffc87f94b4d" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 219.367422ms
8082024-05-30T04:08:53.526ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 26545 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = f89403a7-69d4-4fa5-8581-4e22eb510103 response_code = 500 uri = /cockroachdb
8092024-05-30T04:08:53.526ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f89403a7-69d4-4fa5-8581-4e22eb510103", "content-length": "124", "date": "Thu, 30 May 2024 04:08:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f89403a7-69d4-4fa5-8581-4e22eb510103" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 237.397981ms
8102024-05-30T04:08:53.790ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 26083 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = 71671a25-04b9-4456-9281-b2a33a92788d response_code = 500 uri = /cockroachdb
8112024-05-30T04:08:53.790ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "71671a25-04b9-4456-9281-b2a33a92788d", "content-length": "124", "date": "Thu, 30 May 2024 04:08:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "71671a25-04b9-4456-9281-b2a33a92788d" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 317.164001ms
8122024-05-30T04:08:54.135ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 26737 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = ff892501-18ee-446f-9fdc-db2917fd14ae response_code = 500 uri = /cockroachdb
8132024-05-30T04:08:54.135ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ff892501-18ee-446f-9fdc-db2917fd14ae", "content-length": "124", "date": "Thu, 30 May 2024 04:08:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ff892501-18ee-446f-9fdc-db2917fd14ae" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 185.100816ms
8142024-05-30T04:08:54.351ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 29260 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = 9a6d0f3d-eab1-49af-9588-6f5fdcf1e7f8 response_code = 500 uri = /cockroachdb
8152024-05-30T04:08:54.351ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9a6d0f3d-eab1-49af-9588-6f5fdcf1e7f8", "content-length": "124", "date": "Thu, 30 May 2024 04:08:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9a6d0f3d-eab1-49af-9588-6f5fdcf1e7f8" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 406.050653ms
8162024-05-30T04:08:54.786ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 27867 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = 4cc89a21-5cff-420e-8ab2-13266724919f response_code = 500 uri = /cockroachdb
8172024-05-30T04:08:54.786ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4cc89a21-5cff-420e-8ab2-13266724919f", "content-length": "124", "date": "Thu, 30 May 2024 04:08:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4cc89a21-5cff-420e-8ab2-13266724919f" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 454.144362ms
8182024-05-30T04:08:55.269ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to get address: Failed to get address for name oxControlService5/omicron6 in oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b: Zone execution error: Command [/usr/sbin/zlogin oxz_cockroachdb_1e68cd35-156b-4759-b35d-cd686e7b1b4b /usr/sbin/ipadm show-addr -p -o ADDR oxControlService5/omicron6] executed and failed with status: exit status: 1 stdout: \n stderr: ipadm: Address object not found file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 27537 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = 926cd147-7e3a-47ed-bbc3-01bfad0c28d4 response_code = 500 uri = /cockroachdb
8192024-05-30T04:08:55.269ZWARNSledAgent (RSS): Failed to initialize CockroachDB error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "926cd147-7e3a-47ed-bbc3-01bfad0c28d4", "content-length": "124", "date": "Thu, 30 May 2024 04:08:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "926cd147-7e3a-47ed-bbc3-01bfad0c28d4" } file = sled-agent/src/rack_setup/service.rs:959 retry_after = 482.162635ms
8202024-05-30T04:08:55.780ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::3]:32221 file = sled-agent/src/services.rs:3465
8212024-05-30T04:08:59.368ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3483
8222024-05-30T04:09:11.996ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3506
8232024-05-30T04:09:11.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 16245491 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:51068 req_id = 72a23f32-5b34-4cf4-be90-27abd0e268ec response_code = 204 uri = /cockroachdb
8242024-05-30T04:09:12.001ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.086634968s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
8252024-05-30T04:09:12.031ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
8262024-05-30T04:09:12.031ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51321
8272024-05-30T04:09:12.031ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8282024-05-30T04:09:12.031ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8292024-05-30T04:09:12.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8302024-05-30T04:09:12.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
8312024-05-30T04:09:12.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8322024-05-30T04:09:12.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8332024-05-30T04:09:12.096ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8342024-05-30T04:09:12.096ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8352024-05-30T04:09:12.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8362024-05-30T04:09:12.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8372024-05-30T04:09:12.135ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8382024-05-30T04:09:12.135ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8392024-05-30T04:09:12.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8402024-05-30T04:09:12.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8412024-05-30T04:09:12.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8422024-05-30T04:09:12.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8432024-05-30T04:09:12.190ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8442024-05-30T04:09:12.190ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8452024-05-30T04:09:12.468ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8462024-05-30T04:09:12.468ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8472024-05-30T04:09:12.631ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
8482024-05-30T04:09:12.631ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8492024-05-30T04:09:12.854ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8502024-05-30T04:09:12.854ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
8512024-05-30T04:09:13.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8522024-05-30T04:09:13.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
8532024-05-30T04:09:13.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8542024-05-30T04:09:13.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
8552024-05-30T04:09:13.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8562024-05-30T04:09:13.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
8572024-05-30T04:09:13.511ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8582024-05-30T04:09:13.511ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
8592024-05-30T04:09:13.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8602024-05-30T04:09:13.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
8612024-05-30T04:09:13.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8622024-05-30T04:09:13.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
8632024-05-30T04:09:13.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8642024-05-30T04:09:13.835ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
8652024-05-30T04:09:13.936ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8662024-05-30T04:09:13.936ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
8672024-05-30T04:09:14.056ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8682024-05-30T04:09:14.056ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
8692024-05-30T04:09:14.057ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
8702024-05-30T04:09:14.057ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
8712024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
8722024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
8732024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
8742024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
8752024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
8762024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
8772024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
8782024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
8792024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
8802024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
8812024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
8822024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
8832024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
8842024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
8852024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
8862024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
8872024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
8882024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
8892024-05-30T04:09:14.179ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
8902024-05-30T04:09:14.180ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 2148823 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = cf343831-5101-4fd6-9a91-c97825aeb306 response_code = 500 uri = /omicron-zones
8912024-05-30T04:09:14.180ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cf343831-5101-4fd6-9a91-c97825aeb306", "content-length": "124", "date": "Thu, 30 May 2024 04:09:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cf343831-5101-4fd6-9a91-c97825aeb306" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 84.998887ms sled_address = [fd00:1122:3344:101::1]:12345
8922024-05-30T04:09:14.266ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
8932024-05-30T04:09:14.267ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8942024-05-30T04:09:14.267ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8952024-05-30T04:09:14.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8962024-05-30T04:09:14.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
8972024-05-30T04:09:14.309ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8982024-05-30T04:09:14.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8992024-05-30T04:09:14.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9002024-05-30T04:09:14.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9012024-05-30T04:09:14.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9022024-05-30T04:09:14.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9032024-05-30T04:09:14.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9042024-05-30T04:09:14.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9052024-05-30T04:09:14.386ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9062024-05-30T04:09:14.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9072024-05-30T04:09:14.403ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9082024-05-30T04:09:14.403ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9092024-05-30T04:09:14.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9102024-05-30T04:09:14.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9112024-05-30T04:09:14.440ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9122024-05-30T04:09:14.440ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9132024-05-30T04:09:14.459ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
9142024-05-30T04:09:14.459ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9152024-05-30T04:09:14.477ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9162024-05-30T04:09:14.477ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
9172024-05-30T04:09:14.482ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52784
9182024-05-30T04:09:14.482ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 80 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52784 req_id = 41666991-fa61-479a-a229-4255c3921993 response_code = 200 uri = /baseboard
9192024-05-30T04:09:14.497ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9202024-05-30T04:09:14.497ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
9212024-05-30T04:09:14.515ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9222024-05-30T04:09:14.515ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
9232024-05-30T04:09:14.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9242024-05-30T04:09:14.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
9252024-05-30T04:09:14.552ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9262024-05-30T04:09:14.552ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
9272024-05-30T04:09:14.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9282024-05-30T04:09:14.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
9292024-05-30T04:09:14.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9302024-05-30T04:09:14.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
9312024-05-30T04:09:14.613ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9322024-05-30T04:09:14.613ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
9332024-05-30T04:09:14.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9342024-05-30T04:09:14.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
9352024-05-30T04:09:14.650ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9362024-05-30T04:09:14.650ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
9372024-05-30T04:09:14.650ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
9382024-05-30T04:09:14.650ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9392024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
9402024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
9412024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
9422024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
9432024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
9442024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
9452024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
9462024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
9472024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
9482024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
9492024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
9502024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
9512024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
9522024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
9532024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
9542024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
9552024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
9562024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
9572024-05-30T04:09:14.740ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
9582024-05-30T04:09:14.740ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 473738 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 0aa2e440-a6c4-4eff-85e9-a14efefd2ba0 response_code = 500 uri = /omicron-zones
9592024-05-30T04:09:14.740ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0aa2e440-a6c4-4eff-85e9-a14efefd2ba0", "content-length": "124", "date": "Thu, 30 May 2024 04:09:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0aa2e440-a6c4-4eff-85e9-a14efefd2ba0" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 150.21727ms sled_address = [fd00:1122:3344:101::1]:12345
9602024-05-30T04:09:14.891ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
9612024-05-30T04:09:14.892ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9622024-05-30T04:09:14.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9632024-05-30T04:09:14.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9642024-05-30T04:09:14.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
9652024-05-30T04:09:14.929ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9662024-05-30T04:09:14.929ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9672024-05-30T04:09:14.945ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9682024-05-30T04:09:14.945ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9692024-05-30T04:09:14.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9702024-05-30T04:09:14.962ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9712024-05-30T04:09:14.980ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9722024-05-30T04:09:14.980ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9732024-05-30T04:09:14.997ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9742024-05-30T04:09:14.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9752024-05-30T04:09:15.019ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9762024-05-30T04:09:15.019ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9772024-05-30T04:09:15.039ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9782024-05-30T04:09:15.039ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9792024-05-30T04:09:15.058ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9802024-05-30T04:09:15.058ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9812024-05-30T04:09:15.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
9822024-05-30T04:09:15.077ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9832024-05-30T04:09:15.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9842024-05-30T04:09:15.095ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
9852024-05-30T04:09:15.116ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9862024-05-30T04:09:15.116ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
9872024-05-30T04:09:15.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9882024-05-30T04:09:15.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
9892024-05-30T04:09:15.154ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9902024-05-30T04:09:15.154ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
9912024-05-30T04:09:15.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9922024-05-30T04:09:15.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
9932024-05-30T04:09:15.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9942024-05-30T04:09:15.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
9952024-05-30T04:09:15.214ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9962024-05-30T04:09:15.214ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
9972024-05-30T04:09:15.232ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9982024-05-30T04:09:15.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
9992024-05-30T04:09:15.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10002024-05-30T04:09:15.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
10012024-05-30T04:09:15.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10022024-05-30T04:09:15.270ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
10032024-05-30T04:09:15.270ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
10042024-05-30T04:09:15.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10052024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
10062024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
10072024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
10082024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
10092024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
10102024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
10112024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
10122024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
10132024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
10142024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
10152024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
10162024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
10172024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
10182024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
10192024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
10202024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
10212024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
10222024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
10232024-05-30T04:09:15.361ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
10242024-05-30T04:09:15.361ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 469315 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = d2a1d175-2ab1-4580-875c-453c9fbaf791 response_code = 500 uri = /omicron-zones
10252024-05-30T04:09:15.361ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d2a1d175-2ab1-4580-875c-453c9fbaf791", "content-length": "124", "date": "Thu, 30 May 2024 04:09:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d2a1d175-2ab1-4580-875c-453c9fbaf791" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 203.12384ms sled_address = [fd00:1122:3344:101::1]:12345
10262024-05-30T04:09:15.565ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
10272024-05-30T04:09:15.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10282024-05-30T04:09:15.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10292024-05-30T04:09:15.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10302024-05-30T04:09:15.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
10312024-05-30T04:09:15.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10322024-05-30T04:09:15.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10332024-05-30T04:09:15.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10342024-05-30T04:09:15.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10352024-05-30T04:09:15.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10362024-05-30T04:09:15.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10372024-05-30T04:09:15.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10382024-05-30T04:09:15.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10392024-05-30T04:09:15.683ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10402024-05-30T04:09:15.683ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10412024-05-30T04:09:15.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10422024-05-30T04:09:15.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10432024-05-30T04:09:15.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10442024-05-30T04:09:15.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10452024-05-30T04:09:15.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10462024-05-30T04:09:15.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10472024-05-30T04:09:15.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
10482024-05-30T04:09:15.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10492024-05-30T04:09:15.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10502024-05-30T04:09:15.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
10512024-05-30T04:09:15.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10522024-05-30T04:09:15.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
10532024-05-30T04:09:15.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10542024-05-30T04:09:15.811ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
10552024-05-30T04:09:15.829ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10562024-05-30T04:09:15.829ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
10572024-05-30T04:09:15.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10582024-05-30T04:09:15.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
10592024-05-30T04:09:15.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10602024-05-30T04:09:15.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
10612024-05-30T04:09:15.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10622024-05-30T04:09:15.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
10632024-05-30T04:09:15.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10642024-05-30T04:09:15.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
10652024-05-30T04:09:15.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10662024-05-30T04:09:15.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
10672024-05-30T04:09:15.938ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10682024-05-30T04:09:15.938ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
10692024-05-30T04:09:15.938ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
10702024-05-30T04:09:15.938ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10712024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
10722024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
10732024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
10742024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
10752024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
10762024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
10772024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
10782024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
10792024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
10802024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
10812024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
10822024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
10832024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
10842024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
10852024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
10862024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
10872024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
10882024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
10892024-05-30T04:09:16.028ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
10902024-05-30T04:09:16.028ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 462784 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 1cbb13e3-c2f7-4434-a65d-a3fce082718f response_code = 500 uri = /omicron-zones
10912024-05-30T04:09:16.028ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1cbb13e3-c2f7-4434-a65d-a3fce082718f", "content-length": "124", "date": "Thu, 30 May 2024 04:09:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1cbb13e3-c2f7-4434-a65d-a3fce082718f" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 179.682272ms sled_address = [fd00:1122:3344:101::1]:12345
10922024-05-30T04:09:16.209ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
10932024-05-30T04:09:16.209ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10942024-05-30T04:09:16.209ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10952024-05-30T04:09:16.231ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10962024-05-30T04:09:16.231ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
10972024-05-30T04:09:16.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10982024-05-30T04:09:16.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10992024-05-30T04:09:16.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11002024-05-30T04:09:16.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11012024-05-30T04:09:16.288ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11022024-05-30T04:09:16.288ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11032024-05-30T04:09:16.307ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11042024-05-30T04:09:16.307ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11052024-05-30T04:09:16.325ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11062024-05-30T04:09:16.325ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11072024-05-30T04:09:16.344ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11082024-05-30T04:09:16.344ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11092024-05-30T04:09:16.363ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11102024-05-30T04:09:16.363ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11112024-05-30T04:09:16.382ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11122024-05-30T04:09:16.382ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11132024-05-30T04:09:16.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
11142024-05-30T04:09:16.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11152024-05-30T04:09:16.418ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11162024-05-30T04:09:16.418ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
11172024-05-30T04:09:16.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11182024-05-30T04:09:16.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
11192024-05-30T04:09:16.454ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11202024-05-30T04:09:16.454ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
11212024-05-30T04:09:16.471ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11222024-05-30T04:09:16.471ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
11232024-05-30T04:09:16.489ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11242024-05-30T04:09:16.489ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
11252024-05-30T04:09:16.506ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11262024-05-30T04:09:16.506ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
11272024-05-30T04:09:16.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11282024-05-30T04:09:16.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
11292024-05-30T04:09:16.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11302024-05-30T04:09:16.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
11312024-05-30T04:09:16.567ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11322024-05-30T04:09:16.567ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
11332024-05-30T04:09:16.587ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11342024-05-30T04:09:16.587ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
11352024-05-30T04:09:16.587ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
11362024-05-30T04:09:16.588ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11372024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
11382024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
11392024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
11402024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
11412024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
11422024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
11432024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
11442024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
11452024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
11462024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
11472024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
11482024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
11492024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
11502024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
11512024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
11522024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
11532024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
11542024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
11552024-05-30T04:09:16.679ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
11562024-05-30T04:09:16.679ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 470515 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 0ef81904-7474-4c1d-b1e1-6058599b167c response_code = 500 uri = /omicron-zones
11572024-05-30T04:09:16.680ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0ef81904-7474-4c1d-b1e1-6058599b167c", "content-length": "124", "date": "Thu, 30 May 2024 04:09:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0ef81904-7474-4c1d-b1e1-6058599b167c" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 200.648582ms sled_address = [fd00:1122:3344:101::1]:12345
11582024-05-30T04:09:16.881ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
11592024-05-30T04:09:16.881ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11602024-05-30T04:09:16.881ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11612024-05-30T04:09:16.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11622024-05-30T04:09:16.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
11632024-05-30T04:09:16.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11642024-05-30T04:09:16.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11652024-05-30T04:09:16.941ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11662024-05-30T04:09:16.941ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11672024-05-30T04:09:16.960ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11682024-05-30T04:09:16.960ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11692024-05-30T04:09:16.978ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11702024-05-30T04:09:16.978ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11712024-05-30T04:09:16.997ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11722024-05-30T04:09:16.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11732024-05-30T04:09:17.017ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11742024-05-30T04:09:17.017ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11752024-05-30T04:09:17.035ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11762024-05-30T04:09:17.035ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11772024-05-30T04:09:17.055ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11782024-05-30T04:09:17.055ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11792024-05-30T04:09:17.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
11802024-05-30T04:09:17.074ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11812024-05-30T04:09:17.093ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11822024-05-30T04:09:17.093ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
11832024-05-30T04:09:17.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11842024-05-30T04:09:17.112ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
11852024-05-30T04:09:17.130ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11862024-05-30T04:09:17.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
11872024-05-30T04:09:17.150ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11882024-05-30T04:09:17.150ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
11892024-05-30T04:09:17.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11902024-05-30T04:09:17.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
11912024-05-30T04:09:17.187ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11922024-05-30T04:09:17.188ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
11932024-05-30T04:09:17.206ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11942024-05-30T04:09:17.206ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
11952024-05-30T04:09:17.225ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11962024-05-30T04:09:17.225ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
11972024-05-30T04:09:17.246ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11982024-05-30T04:09:17.246ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
11992024-05-30T04:09:17.265ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12002024-05-30T04:09:17.265ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
12012024-05-30T04:09:17.265ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
12022024-05-30T04:09:17.265ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12032024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
12042024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
12052024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
12062024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
12072024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
12082024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
12092024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
12102024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
12112024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
12122024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
12132024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
12142024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
12152024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
12162024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
12172024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
12182024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
12192024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
12202024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
12212024-05-30T04:09:17.359ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
12222024-05-30T04:09:17.359ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 478352 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 186a4d81-c06d-4334-8e43-e54c20275122 response_code = 500 uri = /omicron-zones
12232024-05-30T04:09:17.359ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "186a4d81-c06d-4334-8e43-e54c20275122", "content-length": "124", "date": "Thu, 30 May 2024 04:09:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "186a4d81-c06d-4334-8e43-e54c20275122" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 169.481142ms sled_address = [fd00:1122:3344:101::1]:12345
12242024-05-30T04:09:17.530ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
12252024-05-30T04:09:17.530ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12262024-05-30T04:09:17.530ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12272024-05-30T04:09:17.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12282024-05-30T04:09:17.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
12292024-05-30T04:09:17.571ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12302024-05-30T04:09:17.571ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12312024-05-30T04:09:17.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12322024-05-30T04:09:17.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12332024-05-30T04:09:17.609ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12342024-05-30T04:09:17.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12352024-05-30T04:09:17.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12362024-05-30T04:09:17.628ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12372024-05-30T04:09:17.647ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12382024-05-30T04:09:17.647ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12392024-05-30T04:09:17.666ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12402024-05-30T04:09:17.666ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12412024-05-30T04:09:17.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12422024-05-30T04:09:17.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12432024-05-30T04:09:17.703ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12442024-05-30T04:09:17.703ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12452024-05-30T04:09:17.721ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
12462024-05-30T04:09:17.721ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12472024-05-30T04:09:17.739ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12482024-05-30T04:09:17.739ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
12492024-05-30T04:09:17.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12502024-05-30T04:09:17.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
12512024-05-30T04:09:17.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12522024-05-30T04:09:17.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
12532024-05-30T04:09:17.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12542024-05-30T04:09:17.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
12552024-05-30T04:09:17.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12562024-05-30T04:09:17.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
12572024-05-30T04:09:17.830ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12582024-05-30T04:09:17.830ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
12592024-05-30T04:09:17.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12602024-05-30T04:09:17.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
12612024-05-30T04:09:17.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12622024-05-30T04:09:17.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
12632024-05-30T04:09:17.885ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12642024-05-30T04:09:17.885ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
12652024-05-30T04:09:17.903ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12662024-05-30T04:09:17.904ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
12672024-05-30T04:09:17.904ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
12682024-05-30T04:09:17.904ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12692024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
12702024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
12712024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
12722024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
12732024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
12742024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
12752024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
12762024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
12772024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
12782024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
12792024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
12802024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
12812024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
12822024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
12832024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
12842024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
12852024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
12862024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
12872024-05-30T04:09:18.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
12882024-05-30T04:09:18.000ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 470418 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 1efe2abd-2de7-4429-940d-759916a5ef2f response_code = 500 uri = /omicron-zones
12892024-05-30T04:09:18.001ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1efe2abd-2de7-4429-940d-759916a5ef2f", "content-length": "124", "date": "Thu, 30 May 2024 04:09:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1efe2abd-2de7-4429-940d-759916a5ef2f" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 267.152065ms sled_address = [fd00:1122:3344:101::1]:12345
12902024-05-30T04:09:18.269ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
12912024-05-30T04:09:18.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12922024-05-30T04:09:18.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12932024-05-30T04:09:18.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12942024-05-30T04:09:18.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
12952024-05-30T04:09:18.309ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12962024-05-30T04:09:18.309ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12972024-05-30T04:09:18.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12982024-05-30T04:09:18.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12992024-05-30T04:09:18.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13002024-05-30T04:09:18.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13012024-05-30T04:09:18.365ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13022024-05-30T04:09:18.365ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13032024-05-30T04:09:18.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13042024-05-30T04:09:18.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13052024-05-30T04:09:18.402ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13062024-05-30T04:09:18.402ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13072024-05-30T04:09:18.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13082024-05-30T04:09:18.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13092024-05-30T04:09:18.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13102024-05-30T04:09:18.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13112024-05-30T04:09:18.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
13122024-05-30T04:09:18.461ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13132024-05-30T04:09:18.480ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13142024-05-30T04:09:18.480ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
13152024-05-30T04:09:18.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13162024-05-30T04:09:18.498ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
13172024-05-30T04:09:18.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13182024-05-30T04:09:18.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
13192024-05-30T04:09:18.539ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13202024-05-30T04:09:18.539ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
13212024-05-30T04:09:18.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13222024-05-30T04:09:18.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
13232024-05-30T04:09:18.578ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13242024-05-30T04:09:18.578ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
13252024-05-30T04:09:18.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13262024-05-30T04:09:18.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
13272024-05-30T04:09:18.618ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13282024-05-30T04:09:18.618ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
13292024-05-30T04:09:18.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13302024-05-30T04:09:18.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
13312024-05-30T04:09:18.658ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13322024-05-30T04:09:18.658ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
13332024-05-30T04:09:18.659ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
13342024-05-30T04:09:18.659ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13352024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
13362024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
13372024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
13382024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
13392024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
13402024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
13412024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
13422024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
13432024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
13442024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
13452024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
13462024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
13472024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
13482024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
13492024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
13502024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
13512024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
13522024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
13532024-05-30T04:09:18.756ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
13542024-05-30T04:09:18.756ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 487467 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = cc07fb8c-c72a-4827-b33a-ad37465eeaae response_code = 500 uri = /omicron-zones
13552024-05-30T04:09:18.757ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cc07fb8c-c72a-4827-b33a-ad37465eeaae", "content-length": "124", "date": "Thu, 30 May 2024 04:09:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cc07fb8c-c72a-4827-b33a-ad37465eeaae" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 379.440254ms sled_address = [fd00:1122:3344:101::1]:12345
13562024-05-30T04:09:19.137ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
13572024-05-30T04:09:19.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13582024-05-30T04:09:19.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13592024-05-30T04:09:19.158ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13602024-05-30T04:09:19.158ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
13612024-05-30T04:09:19.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13622024-05-30T04:09:19.177ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13632024-05-30T04:09:19.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13642024-05-30T04:09:19.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13652024-05-30T04:09:19.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13662024-05-30T04:09:19.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13672024-05-30T04:09:19.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13682024-05-30T04:09:19.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13692024-05-30T04:09:19.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13702024-05-30T04:09:19.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13712024-05-30T04:09:19.273ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13722024-05-30T04:09:19.273ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13732024-05-30T04:09:19.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13742024-05-30T04:09:19.291ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13752024-05-30T04:09:19.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13762024-05-30T04:09:19.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13772024-05-30T04:09:19.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
13782024-05-30T04:09:19.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13792024-05-30T04:09:19.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13802024-05-30T04:09:19.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
13812024-05-30T04:09:19.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13822024-05-30T04:09:19.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
13832024-05-30T04:09:19.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13842024-05-30T04:09:19.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
13852024-05-30T04:09:19.404ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13862024-05-30T04:09:19.404ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
13872024-05-30T04:09:19.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13882024-05-30T04:09:19.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
13892024-05-30T04:09:19.443ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13902024-05-30T04:09:19.443ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
13912024-05-30T04:09:19.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13922024-05-30T04:09:19.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
13932024-05-30T04:09:19.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13942024-05-30T04:09:19.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
13952024-05-30T04:09:19.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13962024-05-30T04:09:19.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
13972024-05-30T04:09:19.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13982024-05-30T04:09:19.524ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
13992024-05-30T04:09:19.524ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
14002024-05-30T04:09:19.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14012024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
14022024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
14032024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
14042024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
14052024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
14062024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
14072024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
14082024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
14092024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
14102024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
14112024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
14122024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
14132024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
14142024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
14152024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
14162024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
14172024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
14182024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
14192024-05-30T04:09:19.617ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
14202024-05-30T04:09:19.617ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 479758 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 2aef4c5b-2d82-4384-bdf5-26322225091b response_code = 500 uri = /omicron-zones
14212024-05-30T04:09:19.617ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2aef4c5b-2d82-4384-bdf5-26322225091b", "content-length": "124", "date": "Thu, 30 May 2024 04:09:19 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2aef4c5b-2d82-4384-bdf5-26322225091b" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 574.977829ms sled_address = [fd00:1122:3344:101::1]:12345
14222024-05-30T04:09:20.193ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
14232024-05-30T04:09:20.193ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14242024-05-30T04:09:20.193ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14252024-05-30T04:09:20.215ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14262024-05-30T04:09:20.215ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
14272024-05-30T04:09:20.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14282024-05-30T04:09:20.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14292024-05-30T04:09:20.254ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14302024-05-30T04:09:20.254ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14312024-05-30T04:09:20.273ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14322024-05-30T04:09:20.273ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14332024-05-30T04:09:20.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14342024-05-30T04:09:20.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14352024-05-30T04:09:20.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14362024-05-30T04:09:20.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14372024-05-30T04:09:20.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14382024-05-30T04:09:20.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14392024-05-30T04:09:20.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14402024-05-30T04:09:20.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14412024-05-30T04:09:20.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14422024-05-30T04:09:20.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14432024-05-30T04:09:20.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
14442024-05-30T04:09:20.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14452024-05-30T04:09:20.405ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14462024-05-30T04:09:20.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
14472024-05-30T04:09:20.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14482024-05-30T04:09:20.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
14492024-05-30T04:09:20.442ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14502024-05-30T04:09:20.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
14512024-05-30T04:09:20.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14522024-05-30T04:09:20.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
14532024-05-30T04:09:20.484ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14542024-05-30T04:09:20.484ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
14552024-05-30T04:09:20.503ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14562024-05-30T04:09:20.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
14572024-05-30T04:09:20.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14582024-05-30T04:09:20.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
14592024-05-30T04:09:20.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14602024-05-30T04:09:20.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
14612024-05-30T04:09:20.562ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14622024-05-30T04:09:20.562ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
14632024-05-30T04:09:20.580ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14642024-05-30T04:09:20.580ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
14652024-05-30T04:09:20.581ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
14662024-05-30T04:09:20.581ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14672024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
14682024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
14692024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
14702024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
14712024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
14722024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
14732024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
14742024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
14752024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
14762024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
14772024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
14782024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
14792024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
14802024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
14812024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
14822024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
14832024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
14842024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
14852024-05-30T04:09:20.678ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
14862024-05-30T04:09:20.678ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 485769 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 61d5559e-2423-4fe8-a79c-130e8adba4d1 response_code = 500 uri = /omicron-zones
14872024-05-30T04:09:20.679ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "61d5559e-2423-4fe8-a79c-130e8adba4d1", "content-length": "124", "date": "Thu, 30 May 2024 04:09:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "61d5559e-2423-4fe8-a79c-130e8adba4d1" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 685.969766ms sled_address = [fd00:1122:3344:101::1]:12345
14882024-05-30T04:09:21.366ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
14892024-05-30T04:09:21.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14902024-05-30T04:09:21.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14912024-05-30T04:09:21.389ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14922024-05-30T04:09:21.389ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
14932024-05-30T04:09:21.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14942024-05-30T04:09:21.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14952024-05-30T04:09:21.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14962024-05-30T04:09:21.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14972024-05-30T04:09:21.451ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14982024-05-30T04:09:21.451ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14992024-05-30T04:09:21.471ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15002024-05-30T04:09:21.471ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15012024-05-30T04:09:21.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15022024-05-30T04:09:21.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15032024-05-30T04:09:21.512ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15042024-05-30T04:09:21.512ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15052024-05-30T04:09:21.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15062024-05-30T04:09:21.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15072024-05-30T04:09:21.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15082024-05-30T04:09:21.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15092024-05-30T04:09:21.574ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
15102024-05-30T04:09:21.574ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15112024-05-30T04:09:21.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15122024-05-30T04:09:21.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
15132024-05-30T04:09:21.613ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15142024-05-30T04:09:21.613ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
15152024-05-30T04:09:21.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15162024-05-30T04:09:21.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
15172024-05-30T04:09:21.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15182024-05-30T04:09:21.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
15192024-05-30T04:09:21.677ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15202024-05-30T04:09:21.677ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
15212024-05-30T04:09:21.697ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15222024-05-30T04:09:21.697ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
15232024-05-30T04:09:21.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15242024-05-30T04:09:21.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
15252024-05-30T04:09:21.735ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15262024-05-30T04:09:21.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
15272024-05-30T04:09:21.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15282024-05-30T04:09:21.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
15292024-05-30T04:09:21.773ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15302024-05-30T04:09:21.773ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
15312024-05-30T04:09:21.773ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
15322024-05-30T04:09:21.773ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15332024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
15342024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
15352024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
15362024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
15372024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
15382024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
15392024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
15402024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
15412024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
15422024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
15432024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
15442024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
15452024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
15462024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
15472024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
15482024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
15492024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
15502024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
15512024-05-30T04:09:21.873ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
15522024-05-30T04:09:21.873ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 507384 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = bca1d3de-a51d-465f-8ce9-80e0970581b5 response_code = 500 uri = /omicron-zones
15532024-05-30T04:09:21.874ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "bca1d3de-a51d-465f-8ce9-80e0970581b5", "content-length": "124", "date": "Thu, 30 May 2024 04:09:21 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "bca1d3de-a51d-465f-8ce9-80e0970581b5" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 482.86218ms sled_address = [fd00:1122:3344:101::1]:12345
15542024-05-30T04:09:22.358ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
15552024-05-30T04:09:22.359ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15562024-05-30T04:09:22.359ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15572024-05-30T04:09:22.384ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15582024-05-30T04:09:22.384ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
15592024-05-30T04:09:22.407ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15602024-05-30T04:09:22.407ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15612024-05-30T04:09:22.427ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15622024-05-30T04:09:22.427ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15632024-05-30T04:09:22.446ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15642024-05-30T04:09:22.446ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15652024-05-30T04:09:22.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15662024-05-30T04:09:22.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15672024-05-30T04:09:22.485ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15682024-05-30T04:09:22.485ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15692024-05-30T04:09:22.504ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15702024-05-30T04:09:22.504ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15712024-05-30T04:09:22.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15722024-05-30T04:09:22.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15732024-05-30T04:09:22.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15742024-05-30T04:09:22.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15752024-05-30T04:09:22.563ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
15762024-05-30T04:09:22.563ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15772024-05-30T04:09:22.582ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15782024-05-30T04:09:22.582ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
15792024-05-30T04:09:22.602ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15802024-05-30T04:09:22.602ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
15812024-05-30T04:09:22.620ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15822024-05-30T04:09:22.620ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
15832024-05-30T04:09:22.641ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15842024-05-30T04:09:22.641ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
15852024-05-30T04:09:22.661ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15862024-05-30T04:09:22.661ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
15872024-05-30T04:09:22.681ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15882024-05-30T04:09:22.681ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
15892024-05-30T04:09:22.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15902024-05-30T04:09:22.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
15912024-05-30T04:09:22.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15922024-05-30T04:09:22.718ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
15932024-05-30T04:09:22.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15942024-05-30T04:09:22.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
15952024-05-30T04:09:22.757ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15962024-05-30T04:09:22.758ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
15972024-05-30T04:09:22.758ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
15982024-05-30T04:09:22.758ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15992024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
16002024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
16012024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
16022024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
16032024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
16042024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
16052024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
16062024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
16072024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
16082024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
16092024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
16102024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
16112024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
16122024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
16132024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
16142024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
16152024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
16162024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
16172024-05-30T04:09:22.852ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
16182024-05-30T04:09:22.852ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 493815 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 9ec68752-d1d5-454b-b739-40566188b006 response_code = 500 uri = /omicron-zones
16192024-05-30T04:09:22.852ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9ec68752-d1d5-454b-b739-40566188b006", "content-length": "124", "date": "Thu, 30 May 2024 04:09:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9ec68752-d1d5-454b-b739-40566188b006" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 650.68261ms sled_address = [fd00:1122:3344:101::1]:12345
16202024-05-30T04:09:23.504ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
16212024-05-30T04:09:23.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16222024-05-30T04:09:23.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16232024-05-30T04:09:23.529ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16242024-05-30T04:09:23.529ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
16252024-05-30T04:09:23.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16262024-05-30T04:09:23.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16272024-05-30T04:09:23.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16282024-05-30T04:09:23.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16292024-05-30T04:09:23.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16302024-05-30T04:09:23.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16312024-05-30T04:09:23.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16322024-05-30T04:09:23.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16332024-05-30T04:09:23.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16342024-05-30T04:09:23.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16352024-05-30T04:09:23.646ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16362024-05-30T04:09:23.646ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16372024-05-30T04:09:23.666ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16382024-05-30T04:09:23.666ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16392024-05-30T04:09:23.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16402024-05-30T04:09:23.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16412024-05-30T04:09:23.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
16422024-05-30T04:09:23.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16432024-05-30T04:09:23.729ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16442024-05-30T04:09:23.729ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
16452024-05-30T04:09:23.750ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16462024-05-30T04:09:23.750ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
16472024-05-30T04:09:23.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16482024-05-30T04:09:23.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
16492024-05-30T04:09:23.789ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16502024-05-30T04:09:23.789ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
16512024-05-30T04:09:23.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16522024-05-30T04:09:23.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
16532024-05-30T04:09:23.830ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16542024-05-30T04:09:23.830ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
16552024-05-30T04:09:23.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16562024-05-30T04:09:23.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
16572024-05-30T04:09:23.868ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16582024-05-30T04:09:23.868ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
16592024-05-30T04:09:23.887ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16602024-05-30T04:09:23.887ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
16612024-05-30T04:09:23.907ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16622024-05-30T04:09:23.907ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
16632024-05-30T04:09:23.907ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
16642024-05-30T04:09:23.908ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16652024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
16662024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
16672024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
16682024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
16692024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
16702024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
16712024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
16722024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
16732024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
16742024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
16752024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
16762024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
16772024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
16782024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
16792024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
16802024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
16812024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
16822024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
16832024-05-30T04:09:24.003ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
16842024-05-30T04:09:24.003ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 499281 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = b36a1902-2e61-40ca-ac07-02c669aae364 response_code = 500 uri = /omicron-zones
16852024-05-30T04:09:24.004ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b36a1902-2e61-40ca-ac07-02c669aae364", "content-length": "124", "date": "Thu, 30 May 2024 04:09:23 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b36a1902-2e61-40ca-ac07-02c669aae364" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 599.161842ms sled_address = [fd00:1122:3344:101::1]:12345
16862024-05-30T04:09:24.604ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
16872024-05-30T04:09:24.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16882024-05-30T04:09:24.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16892024-05-30T04:09:24.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16902024-05-30T04:09:24.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
16912024-05-30T04:09:24.647ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16922024-05-30T04:09:24.647ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16932024-05-30T04:09:24.667ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16942024-05-30T04:09:24.667ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16952024-05-30T04:09:24.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16962024-05-30T04:09:24.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16972024-05-30T04:09:24.707ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16982024-05-30T04:09:24.707ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16992024-05-30T04:09:24.725ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17002024-05-30T04:09:24.725ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17012024-05-30T04:09:24.742ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17022024-05-30T04:09:24.742ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17032024-05-30T04:09:24.760ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17042024-05-30T04:09:24.760ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17052024-05-30T04:09:24.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17062024-05-30T04:09:24.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17072024-05-30T04:09:24.802ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
17082024-05-30T04:09:24.802ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17092024-05-30T04:09:24.822ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17102024-05-30T04:09:24.822ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
17112024-05-30T04:09:24.841ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17122024-05-30T04:09:24.841ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
17132024-05-30T04:09:24.860ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17142024-05-30T04:09:24.860ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
17152024-05-30T04:09:24.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17162024-05-30T04:09:24.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
17172024-05-30T04:09:24.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17182024-05-30T04:09:24.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17192024-05-30T04:09:24.918ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17202024-05-30T04:09:24.918ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
17212024-05-30T04:09:24.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17222024-05-30T04:09:24.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17232024-05-30T04:09:24.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17242024-05-30T04:09:24.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
17252024-05-30T04:09:24.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17262024-05-30T04:09:24.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
17272024-05-30T04:09:24.997ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17282024-05-30T04:09:24.997ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
17292024-05-30T04:09:24.997ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
17302024-05-30T04:09:24.997ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17312024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
17322024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
17332024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
17342024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
17352024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
17362024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
17372024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
17382024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
17392024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
17402024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
17412024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
17422024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
17432024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
17442024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
17452024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
17462024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
17472024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
17482024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
17492024-05-30T04:09:25.095ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
17502024-05-30T04:09:25.095ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 490961 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 814e3ddc-a4b7-48a1-8b5d-cb88abf0cca6 response_code = 500 uri = /omicron-zones
17512024-05-30T04:09:25.095ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "814e3ddc-a4b7-48a1-8b5d-cb88abf0cca6", "content-length": "124", "date": "Thu, 30 May 2024 04:09:25 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "814e3ddc-a4b7-48a1-8b5d-cb88abf0cca6" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.264639866s sled_address = [fd00:1122:3344:101::1]:12345
17522024-05-30T04:09:26.362ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
17532024-05-30T04:09:26.362ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17542024-05-30T04:09:26.362ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17552024-05-30T04:09:26.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17562024-05-30T04:09:26.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
17572024-05-30T04:09:26.404ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17582024-05-30T04:09:26.404ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17592024-05-30T04:09:26.425ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17602024-05-30T04:09:26.425ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17612024-05-30T04:09:26.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17622024-05-30T04:09:26.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17632024-05-30T04:09:26.463ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17642024-05-30T04:09:26.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17652024-05-30T04:09:26.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17662024-05-30T04:09:26.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17672024-05-30T04:09:26.500ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17682024-05-30T04:09:26.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17692024-05-30T04:09:26.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17702024-05-30T04:09:26.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17712024-05-30T04:09:26.546ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17722024-05-30T04:09:26.546ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17732024-05-30T04:09:26.568ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
17742024-05-30T04:09:26.568ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17752024-05-30T04:09:26.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17762024-05-30T04:09:26.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
17772024-05-30T04:09:26.610ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17782024-05-30T04:09:26.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
17792024-05-30T04:09:26.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17802024-05-30T04:09:26.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
17812024-05-30T04:09:26.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17822024-05-30T04:09:26.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
17832024-05-30T04:09:26.665ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17842024-05-30T04:09:26.665ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17852024-05-30T04:09:26.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17862024-05-30T04:09:26.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
17872024-05-30T04:09:26.706ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17882024-05-30T04:09:26.706ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
17892024-05-30T04:09:26.727ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17902024-05-30T04:09:26.727ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
17912024-05-30T04:09:26.746ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17922024-05-30T04:09:26.747ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
17932024-05-30T04:09:26.767ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17942024-05-30T04:09:26.767ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
17952024-05-30T04:09:26.767ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
17962024-05-30T04:09:26.767ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17972024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
17982024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
17992024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
18002024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
18012024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
18022024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
18032024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
18042024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
18052024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
18062024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
18072024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
18082024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
18092024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
18102024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
18112024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
18122024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
18132024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
18142024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
18152024-05-30T04:09:26.863ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
18162024-05-30T04:09:26.863ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 501219 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 6ec97c01-c043-415b-821a-9c8cc67ab73f response_code = 500 uri = /omicron-zones
18172024-05-30T04:09:26.863ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6ec97c01-c043-415b-821a-9c8cc67ab73f", "content-length": "124", "date": "Thu, 30 May 2024 04:09:26 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6ec97c01-c043-415b-821a-9c8cc67ab73f" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 932.009198ms sled_address = [fd00:1122:3344:101::1]:12345
18182024-05-30T04:09:27.796ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
18192024-05-30T04:09:27.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18202024-05-30T04:09:27.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18212024-05-30T04:09:27.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18222024-05-30T04:09:27.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
18232024-05-30T04:09:27.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18242024-05-30T04:09:27.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18252024-05-30T04:09:27.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18262024-05-30T04:09:27.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18272024-05-30T04:09:27.874ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18282024-05-30T04:09:27.874ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18292024-05-30T04:09:27.894ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18302024-05-30T04:09:27.894ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18312024-05-30T04:09:27.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18322024-05-30T04:09:27.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18332024-05-30T04:09:27.931ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18342024-05-30T04:09:27.931ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18352024-05-30T04:09:27.952ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18362024-05-30T04:09:27.952ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18372024-05-30T04:09:27.971ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18382024-05-30T04:09:27.971ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18392024-05-30T04:09:27.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
18402024-05-30T04:09:27.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18412024-05-30T04:09:28.013ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18422024-05-30T04:09:28.013ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
18432024-05-30T04:09:28.034ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18442024-05-30T04:09:28.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
18452024-05-30T04:09:28.056ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18462024-05-30T04:09:28.056ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
18472024-05-30T04:09:28.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18482024-05-30T04:09:28.075ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
18492024-05-30T04:09:28.094ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18502024-05-30T04:09:28.094ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
18512024-05-30T04:09:28.114ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18522024-05-30T04:09:28.114ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
18532024-05-30T04:09:28.122ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.304142322s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
18542024-05-30T04:09:28.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18552024-05-30T04:09:28.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
18562024-05-30T04:09:28.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18572024-05-30T04:09:28.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
18582024-05-30T04:09:28.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18592024-05-30T04:09:28.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
18602024-05-30T04:09:28.193ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18612024-05-30T04:09:28.193ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
18622024-05-30T04:09:28.193ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
18632024-05-30T04:09:28.193ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18642024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
18652024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
18662024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
18672024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
18682024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
18692024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
18702024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
18712024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
18722024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
18732024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
18742024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
18752024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
18762024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
18772024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
18782024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
18792024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
18802024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
18812024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
18822024-05-30T04:09:28.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
18832024-05-30T04:09:28.289ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 492378 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 4132997f-98c9-4ee9-84c1-79273c7467c9 response_code = 500 uri = /omicron-zones
18842024-05-30T04:09:28.289ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4132997f-98c9-4ee9-84c1-79273c7467c9", "content-length": "124", "date": "Thu, 30 May 2024 04:09:27 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4132997f-98c9-4ee9-84c1-79273c7467c9" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 939.279519ms sled_address = [fd00:1122:3344:101::1]:12345
18852024-05-30T04:09:29.229ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
18862024-05-30T04:09:29.229ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18872024-05-30T04:09:29.229ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18882024-05-30T04:09:29.252ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18892024-05-30T04:09:29.252ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
18902024-05-30T04:09:29.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18912024-05-30T04:09:29.271ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18922024-05-30T04:09:29.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18932024-05-30T04:09:29.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18942024-05-30T04:09:29.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18952024-05-30T04:09:29.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18962024-05-30T04:09:29.334ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18972024-05-30T04:09:29.334ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18982024-05-30T04:09:29.355ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18992024-05-30T04:09:29.355ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19002024-05-30T04:09:29.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19012024-05-30T04:09:29.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19022024-05-30T04:09:29.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19032024-05-30T04:09:29.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19042024-05-30T04:09:29.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19052024-05-30T04:09:29.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19062024-05-30T04:09:29.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
19072024-05-30T04:09:29.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19082024-05-30T04:09:29.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19092024-05-30T04:09:29.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
19102024-05-30T04:09:29.478ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19112024-05-30T04:09:29.478ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
19122024-05-30T04:09:29.499ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19132024-05-30T04:09:29.500ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
19142024-05-30T04:09:29.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19152024-05-30T04:09:29.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
19162024-05-30T04:09:29.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19172024-05-30T04:09:29.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
19182024-05-30T04:09:29.562ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19192024-05-30T04:09:29.562ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
19202024-05-30T04:09:29.582ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19212024-05-30T04:09:29.582ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
19222024-05-30T04:09:29.602ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19232024-05-30T04:09:29.602ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
19242024-05-30T04:09:29.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19252024-05-30T04:09:29.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
19262024-05-30T04:09:29.640ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19272024-05-30T04:09:29.640ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
19282024-05-30T04:09:29.640ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
19292024-05-30T04:09:29.641ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19302024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
19312024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
19322024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
19332024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
19342024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
19352024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
19362024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
19372024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
19382024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
19392024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
19402024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
19412024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
19422024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
19432024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
19442024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
19452024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
19462024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
19472024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
19482024-05-30T04:09:29.739ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
19492024-05-30T04:09:29.739ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 509867 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 507e4c08-17b0-4607-8bfc-672a3358f93c response_code = 500 uri = /omicron-zones
19502024-05-30T04:09:29.739ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "507e4c08-17b0-4607-8bfc-672a3358f93c", "content-length": "124", "date": "Thu, 30 May 2024 04:09:29 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "507e4c08-17b0-4607-8bfc-672a3358f93c" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.39848719s sled_address = [fd00:1122:3344:101::1]:12345
19512024-05-30T04:09:31.139ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
19522024-05-30T04:09:31.140ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19532024-05-30T04:09:31.140ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19542024-05-30T04:09:31.163ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19552024-05-30T04:09:31.163ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
19562024-05-30T04:09:31.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19572024-05-30T04:09:31.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19582024-05-30T04:09:31.206ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19592024-05-30T04:09:31.206ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19602024-05-30T04:09:31.226ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19612024-05-30T04:09:31.226ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19622024-05-30T04:09:31.245ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19632024-05-30T04:09:31.245ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19642024-05-30T04:09:31.264ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19652024-05-30T04:09:31.264ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19662024-05-30T04:09:31.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19672024-05-30T04:09:31.283ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19682024-05-30T04:09:31.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19692024-05-30T04:09:31.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19702024-05-30T04:09:31.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19712024-05-30T04:09:31.321ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19722024-05-30T04:09:31.341ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
19732024-05-30T04:09:31.341ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19742024-05-30T04:09:31.360ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19752024-05-30T04:09:31.360ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
19762024-05-30T04:09:31.379ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19772024-05-30T04:09:31.379ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
19782024-05-30T04:09:31.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19792024-05-30T04:09:31.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
19802024-05-30T04:09:31.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19812024-05-30T04:09:31.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
19822024-05-30T04:09:31.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19832024-05-30T04:09:31.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
19842024-05-30T04:09:31.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19852024-05-30T04:09:31.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
19862024-05-30T04:09:31.478ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19872024-05-30T04:09:31.478ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
19882024-05-30T04:09:31.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19892024-05-30T04:09:31.498ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
19902024-05-30T04:09:31.517ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19912024-05-30T04:09:31.517ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
19922024-05-30T04:09:31.540ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19932024-05-30T04:09:31.540ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
19942024-05-30T04:09:31.540ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
19952024-05-30T04:09:31.540ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19962024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
19972024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
19982024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
19992024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
20002024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
20012024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
20022024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
20032024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
20042024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
20052024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
20062024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
20072024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
20082024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
20092024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
20102024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
20112024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
20122024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
20132024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
20142024-05-30T04:09:31.639ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3058 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
20152024-05-30T04:09:31.639ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5", TimeNotSynchronized), ("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232", TimeNotSynchronized), ("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f", TimeNotSynchronized), ("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3", TimeNotSynchronized), ("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668", TimeNotSynchronized), ("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea", TimeNotSynchronized), ("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2", TimeNotSynchronized), ("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba", TimeNotSynchronized), ("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed", TimeNotSynchronized), ("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48", TimeNotSynchronized), ("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144", TimeNotSynchronized), ("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c", TimeNotSynchronized), ("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128", TimeNotSynchronized), ("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1", TimeNotSynchronized), ("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab", TimeNotSynchronized), ("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4", TimeNotSynchronized), ("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f", TimeNotSynchronized), ("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835", TimeNotSynchronized), ("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 499457 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = 0e813884-0e18-4a62-a41f-39d9ccac723b response_code = 500 uri = /omicron-zones
20162024-05-30T04:09:31.639ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0e813884-0e18-4a62-a41f-39d9ccac723b", "content-length": "124", "date": "Thu, 30 May 2024 04:09:31 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0e813884-0e18-4a62-a41f-39d9ccac723b" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 2.004534692s sled_address = [fd00:1122:3344:101::1]:12345
20172024-05-30T04:09:33.007ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 5269a058-580b-4ad2-a686-a68f8b8e0982, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fd14d7c8-e7c7-4706-93f1-737bc0db512a, kind: Service { id: 37aef7a7-53c8-4b7f-aab3-5292349f3f5d }, name: Name("ntp-37aef7a7-53c8-4b7f-aab3-5292349f3f5d"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 65])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: 72c3976d-68e9-463a-be70-3de38abfe035, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: a84ccf3e-ca81-499f-b526-5c900c243456, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9112ad3c-6c15-4601-bf24-48148ee77690, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 5cc3fd7a-881a-45bf-9524-016003ecf668, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 8f3604d6-5231-4728-9097-caf093b736ed, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ca8c551e-b7e2-4add-a459-c77679e168ba, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: c2209168-b7ee-4e88-824f-9effc24609ea, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 2bbcf07c-7614-4e39-9570-91a5c47d5232, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External } } } }, OmicronZoneConfig { id: df8c46fb-39e3-4705-bde9-709842a66be5, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External } } } }, OmicronZoneConfig { id: dbc6937e-eabc-4bd1-922e-b561820024c2, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15faa041-d273-4f44-b56e-98664dd6c144, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External } } } }, OmicronZoneConfig { id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 1f1da100-e042-4583-8561-6eff4c72a835, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 3388152a-babe-4c26-b4bb-14448f905920, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
20182024-05-30T04:09:33.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20192024-05-30T04:09:33.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 5269a058-580b-4ad2-a686-a68f8b8e0982, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20202024-05-30T04:09:33.031ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20212024-05-30T04:09:33.031ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 448a7523-abaa-4f37-8ea0-88fd1a15dcfd, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: InternalDns }, responder: ... }
20222024-05-30T04:09:33.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20232024-05-30T04:09:33.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a5f19f92-dd9d-4d5d-b88b-df21ddc95063, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20242024-05-30T04:09:33.070ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20252024-05-30T04:09:33.070ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1e68cd35-156b-4759-b35d-cd686e7b1b4b, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20262024-05-30T04:09:33.089ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20272024-05-30T04:09:33.089ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: fef621b3-c922-4648-a07b-e45ce7dbe3e0, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20282024-05-30T04:09:33.108ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20292024-05-30T04:09:33.108ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 72c3976d-68e9-463a-be70-3de38abfe035, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20302024-05-30T04:09:33.128ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20312024-05-30T04:09:33.128ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a84ccf3e-ca81-499f-b526-5c900c243456, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20322024-05-30T04:09:33.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20332024-05-30T04:09:33.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9112ad3c-6c15-4601-bf24-48148ee77690, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20342024-05-30T04:09:33.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20352024-05-30T04:09:33.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 56096d63-69cd-4c67-9c8f-fe757a4566d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20362024-05-30T04:09:33.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20372024-05-30T04:09:33.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20382024-05-30T04:09:33.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
20392024-05-30T04:09:33.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: ca8c551e-b7e2-4add-a459-c77679e168ba, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20402024-05-30T04:09:33.222ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20412024-05-30T04:09:33.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b5ac113d-4f55-4c3e-8718-a1f7414bba5f, dataset_name: DatasetName { pool_name: ZpoolName { id: 3022fa0e-6566-4854-a1a9-1e0c5908d5b0 (zpool), kind: External }, kind: Crucible }, responder: ... }
20422024-05-30T04:09:33.240ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20432024-05-30T04:09:33.240ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f5dbf3d-e4c0-438a-82cb-4d43a6453128, dataset_name: DatasetName { pool_name: ZpoolName { id: 1793c88e-7478-4741-8254-860fae35146b (zpool), kind: External }, kind: Crucible }, responder: ... }
20442024-05-30T04:09:33.257ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20452024-05-30T04:09:33.257ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: df8c46fb-39e3-4705-bde9-709842a66be5, dataset_name: DatasetName { pool_name: ZpoolName { id: f61d361f-e4c0-48df-9191-ff705041ee76 (zpool), kind: External }, kind: Crucible }, responder: ... }
20462024-05-30T04:09:33.275ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20472024-05-30T04:09:33.275ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbc6937e-eabc-4bd1-922e-b561820024c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7 (zpool), kind: External }, kind: Crucible }, responder: ... }
20482024-05-30T04:09:33.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20492024-05-30T04:09:33.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 15faa041-d273-4f44-b56e-98664dd6c144, dataset_name: DatasetName { pool_name: ZpoolName { id: 5d12239f-a80e-4f75-83d4-ebef24b6fea1 (zpool), kind: External }, kind: Crucible }, responder: ... }
20502024-05-30T04:09:33.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20512024-05-30T04:09:33.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 47b0a86a-e796-41a6-af75-fb0dfd4a86c1, dataset_name: DatasetName { pool_name: ZpoolName { id: a168fba6-15db-4743-814f-b76d3daf999f (zpool), kind: External }, kind: Crucible }, responder: ... }
20522024-05-30T04:09:33.329ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20532024-05-30T04:09:33.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: dbfa33e5-1797-4930-9f90-903e48c8cf4c, dataset_name: DatasetName { pool_name: ZpoolName { id: 2efcf3a6-ef7f-48c2-8713-0dd19fe385d1 (zpool), kind: External }, kind: Crucible }, responder: ... }
20542024-05-30T04:09:33.347ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20552024-05-30T04:09:33.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1f1da100-e042-4583-8561-6eff4c72a835, dataset_name: DatasetName { pool_name: ZpoolName { id: dc70dd36-578e-4d7e-9ca2-11c36573efc3 (zpool), kind: External }, kind: Crucible }, responder: ... }
20562024-05-30T04:09:33.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20572024-05-30T04:09:33.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 3388152a-babe-4c26-b4bb-14448f905920, dataset_name: DatasetName { pool_name: ZpoolName { id: 8f547a7a-6177-4ca0-82ce-6646cf05d8a6 (zpool), kind: External }, kind: Crucible }, responder: ... }
20582024-05-30T04:09:33.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20592024-05-30T04:09:33.384ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
20602024-05-30T04:09:33.384ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
20612024-05-30T04:09:33.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20622024-05-30T04:09:33.481ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
20632024-05-30T04:09:33.481ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20642024-05-30T04:09:33.483ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
20652024-05-30T04:09:33.541ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20662024-05-30T04:09:33.543ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20672024-05-30T04:09:33.543ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20682024-05-30T04:09:33.544ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20692024-05-30T04:09:33.544ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20702024-05-30T04:09:33.544ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20712024-05-30T04:09:33.544ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20722024-05-30T04:09:33.748ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
20732024-05-30T04:09:33.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20742024-05-30T04:09:33.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20752024-05-30T04:09:33.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20762024-05-30T04:09:33.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20772024-05-30T04:09:33.750ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
20782024-05-30T04:09:33.763ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:259
20792024-05-30T04:09:33.764ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20802024-05-30T04:09:33.765ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20812024-05-30T04:09:33.765ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab file = illumos-utils/src/zone.rs:315
20822024-05-30T04:09:33.771ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea file = illumos-utils/src/zone.rs:315
20832024-05-30T04:09:33.776ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232 file = illumos-utils/src/zone.rs:315
20842024-05-30T04:09:33.781ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f file = illumos-utils/src/zone.rs:315
20852024-05-30T04:09:33.786ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128 file = illumos-utils/src/zone.rs:315
20862024-05-30T04:09:33.791ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5 file = illumos-utils/src/zone.rs:315
20872024-05-30T04:09:33.795ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2 file = illumos-utils/src/zone.rs:315
20882024-05-30T04:09:33.801ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144 file = illumos-utils/src/zone.rs:315
20892024-05-30T04:09:33.920ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab file = illumos-utils/src/zone.rs:354
20902024-05-30T04:09:33.925ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea file = illumos-utils/src/zone.rs:354
20912024-05-30T04:09:33.930ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232 file = illumos-utils/src/zone.rs:354
20922024-05-30T04:09:33.934ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f file = illumos-utils/src/zone.rs:354
20932024-05-30T04:09:33.938ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128 file = illumos-utils/src/zone.rs:354
20942024-05-30T04:09:33.944ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5 file = illumos-utils/src/zone.rs:354
20952024-05-30T04:09:33.949ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2 file = illumos-utils/src/zone.rs:354
20962024-05-30T04:09:33.954ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144 file = illumos-utils/src/zone.rs:354
20972024-05-30T04:09:33.958ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1 file = illumos-utils/src/zone.rs:315
20982024-05-30T04:09:33.962ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835 file = illumos-utils/src/zone.rs:315
20992024-05-30T04:09:33.968ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920 file = illumos-utils/src/zone.rs:315
21002024-05-30T04:09:33.973ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c file = illumos-utils/src/zone.rs:315
21012024-05-30T04:09:33.979ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba file = illumos-utils/src/zone.rs:315
21022024-05-30T04:09:33.986ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed file = illumos-utils/src/zone.rs:315
21032024-05-30T04:09:33.992ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1 file = illumos-utils/src/zone.rs:354
21042024-05-30T04:09:33.997ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835 file = illumos-utils/src/zone.rs:354
21052024-05-30T04:09:34.002ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920 file = illumos-utils/src/zone.rs:354
21062024-05-30T04:09:34.008ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c file = illumos-utils/src/zone.rs:354
21072024-05-30T04:09:34.015ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba file = illumos-utils/src/zone.rs:354
21082024-05-30T04:09:34.022ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed file = illumos-utils/src/zone.rs:354
21092024-05-30T04:09:34.074ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
21102024-05-30T04:09:34.134ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
21112024-05-30T04:09:34.178ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
21122024-05-30T04:09:34.233ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
21132024-05-30T04:09:34.367ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:276
21142024-05-30T04:09:34.368ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
21152024-05-30T04:09:34.368ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
21162024-05-30T04:09:34.462ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 213, 208]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte1" } }
21172024-05-30T04:09:34.462ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
21182024-05-30T04:09:34.466ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
21192024-05-30T04:09:34.466ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
21202024-05-30T04:09:34.595ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte2", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 248, 105]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte2" } }
21212024-05-30T04:09:34.595ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
21222024-05-30T04:09:34.600ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
21232024-05-30T04:09:34.600ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
21242024-05-30T04:09:34.707ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 229, 173]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte3" } }
21252024-05-30T04:09:34.707ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = external_dns
21262024-05-30T04:09:34.711ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
21272024-05-30T04:09:34.711ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
21282024-05-30T04:09:34.793ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 170, 13]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte4" } }
21292024-05-30T04:09:34.793ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = external_dns
21302024-05-30T04:09:34.795ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:301
21312024-05-30T04:09:34.795ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:193
21322024-05-30T04:09:34.889ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 244, 140]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte5" } }
21332024-05-30T04:09:34.889ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1290 zone_type = nexus
21342024-05-30T04:09:34.890ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
21352024-05-30T04:09:34.890ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
21362024-05-30T04:09:34.890ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
21372024-05-30T04:09:34.890ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
21382024-05-30T04:09:34.890ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
21392024-05-30T04:09:34.890ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
21402024-05-30T04:09:34.891ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
21412024-05-30T04:09:34.894ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21422024-05-30T04:09:34.894ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
21432024-05-30T04:09:34.897ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
21442024-05-30T04:09:34.900ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21452024-05-30T04:09:34.900ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21462024-05-30T04:09:35.008ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
21472024-05-30T04:09:35.008ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
21482024-05-30T04:09:35.008ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
21492024-05-30T04:09:35.008ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
21502024-05-30T04:09:35.008ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f file = illumos-utils/src/zone.rs:315
21512024-05-30T04:09:35.016ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4 file = illumos-utils/src/zone.rs:315
21522024-05-30T04:09:35.025ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
21532024-05-30T04:09:35.029ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
21542024-05-30T04:09:35.032ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21552024-05-30T04:09:35.032ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21562024-05-30T04:09:35.032ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668 file = illumos-utils/src/zone.rs:315
21572024-05-30T04:09:35.103ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f file = illumos-utils/src/zone.rs:354
21582024-05-30T04:09:35.108ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4 file = illumos-utils/src/zone.rs:354
21592024-05-30T04:09:35.114ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668 file = illumos-utils/src/zone.rs:354
21602024-05-30T04:09:35.121ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3 file = illumos-utils/src/zone.rs:315
21612024-05-30T04:09:35.127ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48 file = illumos-utils/src/zone.rs:315
21622024-05-30T04:09:35.182ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3 file = illumos-utils/src/zone.rs:354
21632024-05-30T04:09:35.225ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48 file = illumos-utils/src/zone.rs:354
21642024-05-30T04:09:36.024ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::f"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21652024-05-30T04:09:36.025ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
21662024-05-30T04:09:36.116ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::10"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21672024-05-30T04:09:36.116ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
21682024-05-30T04:09:36.130ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::11"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21692024-05-30T04:09:36.130ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
21702024-05-30T04:09:36.198ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::12"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_3022fa0e-6566-4854-a1a9-1e0c5908d5b0/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="f7ae4ef8-3980-4e2c-a2d2-732356c54a03"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21712024-05-30T04:09:36.198ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
21722024-05-30T04:09:36.303ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::14"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_f61d361f-e4c0-48df-9191-ff705041ee76/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="72de9862-9ff1-4026-9f8e-f3411f2ec73d"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21732024-05-30T04:09:36.303ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
21742024-05-30T04:09:36.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 58.282989ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
21752024-05-30T04:09:36.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 27.182519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
21762024-05-30T04:09:36.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 48.727239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
21772024-05-30T04:09:36.415ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::13"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_1793c88e-7478-4741-8254-860fae35146b/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="32303b1b-e0e5-417c-b316-52cad8cda993"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21782024-05-30T04:09:36.416ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
21792024-05-30T04:09:36.424ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::16"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_5d12239f-a80e-4f75-83d4-ebef24b6fea1/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="7c367b51-ea8c-4493-8fb7-bf4d801731f9"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21802024-05-30T04:09:36.424ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::19"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_dc70dd36-578e-4d7e-9ca2-11c36573efc3/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="b1f44db3-fa49-4279-b183-1ae160abf80a"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21812024-05-30T04:09:36.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 40.703986ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
21822024-05-30T04:09:36.435ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::15"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="787bf736-03e0-4c89-ad7f-8eb26a611b29"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21832024-05-30T04:09:36.435ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
21842024-05-30T04:09:36.440ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
21852024-05-30T04:09:36.454ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 50.847056ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
21862024-05-30T04:09:36.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 110.327381ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
21872024-05-30T04:09:36.463ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
21882024-05-30T04:09:36.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 130.745234ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
21892024-05-30T04:09:36.511ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 100.001701ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
21902024-05-30T04:09:36.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 224.679238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
21912024-05-30T04:09:36.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 25.213985ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
21922024-05-30T04:09:36.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 182.376295ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
21932024-05-30T04:09:36.591ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_8f547a7a-6177-4ca0-82ce-6646cf05d8a6/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="fbd55a4a-0d0b-4162-9c16-03a7148a637f"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21942024-05-30T04:09:36.591ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
21952024-05-30T04:09:36.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 75.569759ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
21962024-05-30T04:09:36.648ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::17"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_a168fba6-15db-4743-814f-b76d3daf999f/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="d4e3c962-0dae-4fa0-b46c-6ddf3cfce553"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21972024-05-30T04:09:36.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 211.67388ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
21982024-05-30T04:09:36.695ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
21992024-05-30T04:09:36.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 226.54715ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
22002024-05-30T04:09:36.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 155.863816ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
22012024-05-30T04:09:36.735ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::18"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="9e4d1480-4cd1-41e1-97da-8bd78cf027db"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
22022024-05-30T04:09:36.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 43.325542ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
22032024-05-30T04:09:36.783ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22042024-05-30T04:09:36.792ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::d"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value="8f3604d6-5231-4728-9097-caf093b736ed"/> <propval type="astring" name="address" value="[fd00:1122:3344:101::d]:12223"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
22052024-05-30T04:09:36.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 362.050078ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
22062024-05-30T04:09:36.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 312.79208ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
22072024-05-30T04:09:36.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 444.615301ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
22082024-05-30T04:09:37.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 140.715183ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
22092024-05-30T04:09:37.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 350.576769ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
22102024-05-30T04:09:37.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 53.574183ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
22112024-05-30T04:09:37.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 46.419762ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
22122024-05-30T04:09:37.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 61.932962ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
22132024-05-30T04:09:37.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 72.729913ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
22142024-05-30T04:09:37.084ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22152024-05-30T04:09:37.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 484.089242ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
22162024-05-30T04:09:37.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 95.699721ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
22172024-05-30T04:09:37.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 123.098452ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
22182024-05-30T04:09:37.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 41.328842ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
22192024-05-30T04:09:37.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 107.815251ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
22202024-05-30T04:09:37.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 139.58482ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
22212024-05-30T04:09:37.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 54.637509ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22222024-05-30T04:09:37.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 496.508519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
22232024-05-30T04:09:37.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 688.249307ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
22242024-05-30T04:09:37.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 740.43603ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
22252024-05-30T04:09:37.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 108.897479ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
22262024-05-30T04:09:37.627ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 1.025746671s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
22272024-05-30T04:09:37.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 172.560933ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
22282024-05-30T04:09:37.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 255.216767ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
22292024-05-30T04:09:37.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 50.701607ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
22302024-05-30T04:09:37.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 276.389493ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
22312024-05-30T04:09:37.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 293.078898ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
22322024-05-30T04:09:37.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 103.942307ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22332024-05-30T04:09:37.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 74.0002ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22342024-05-30T04:09:37.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 860.342893ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
22352024-05-30T04:09:37.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 356.901189ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
22362024-05-30T04:09:37.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 576.959442ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
22372024-05-30T04:09:37.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 148.752316ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
22382024-05-30T04:09:37.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 147.032266ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22392024-05-30T04:09:37.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 942.012413ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
22402024-05-30T04:09:37.981ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 146.0423ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22412024-05-30T04:09:38.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 242.894579ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
22422024-05-30T04:09:38.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 353.70743ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
22432024-05-30T04:09:38.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 436.421104ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
22442024-05-30T04:09:38.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 280.722373ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22452024-05-30T04:09:38.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 510.975187ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
22462024-05-30T04:09:38.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 321.786737ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22472024-05-30T04:09:38.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 1.174166098s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
22482024-05-30T04:09:38.253ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 1.47869135s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
22492024-05-30T04:09:38.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 892.510534ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
22502024-05-30T04:09:38.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 924.993436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
22512024-05-30T04:09:38.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 286.255647ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22522024-05-30T04:09:38.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 1.130938679s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
22532024-05-30T04:09:38.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 941.056349ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22542024-05-30T04:09:38.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 660.637276ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
22552024-05-30T04:09:38.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 967.859365ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
22562024-05-30T04:09:38.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 696.032544ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
22572024-05-30T04:09:38.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 734.535082ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
22582024-05-30T04:09:38.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 422.829444ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22592024-05-30T04:09:38.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 646.48524ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
22602024-05-30T04:09:39.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 1.049583483s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
22612024-05-30T04:09:39.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 1.230344511s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
22622024-05-30T04:09:39.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 895.788092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
22632024-05-30T04:09:39.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 914.384513ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
22642024-05-30T04:09:39.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 1.479500974s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22652024-05-30T04:09:39.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 1.316533214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
22662024-05-30T04:09:39.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 852.680851ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
22672024-05-30T04:09:39.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 663.921197ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
22682024-05-30T04:09:39.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 500.389112ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22692024-05-30T04:09:39.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 938.527215ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
22702024-05-30T04:09:39.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 1.151459267s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
22712024-05-30T04:09:39.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 1.269967048s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
22722024-05-30T04:09:39.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 538.520673ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
22732024-05-30T04:09:40.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 653.260749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22742024-05-30T04:09:40.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 1.390240281s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
22752024-05-30T04:09:40.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 1.12831668s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
22762024-05-30T04:09:40.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 1.475465615s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
22772024-05-30T04:09:40.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 577.398411ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
22782024-05-30T04:09:40.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 1.310004438s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
22792024-05-30T04:09:40.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 831.907573ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
22802024-05-30T04:09:40.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 1.123647062s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
22812024-05-30T04:09:40.502ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 1.401910087s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
22822024-05-30T04:09:40.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 1.233907356s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
22832024-05-30T04:09:40.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 1.499597834s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
22842024-05-30T04:09:40.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 834.604763ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
22852024-05-30T04:09:40.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 999.732493ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
22862024-05-30T04:09:40.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 830.228664ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
22872024-05-30T04:09:40.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 658.211269ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22882024-05-30T04:09:40.979ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::e"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
22892024-05-30T04:09:40.979ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
22902024-05-30T04:09:41.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 570.909272ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
22912024-05-30T04:09:41.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 805.271274ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
22922024-05-30T04:09:41.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 64.437485ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
22932024-05-30T04:09:41.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 90.983006ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
22942024-05-30T04:09:41.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 1.072446456s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
22952024-05-30T04:09:41.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 129.972654ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
22962024-05-30T04:09:41.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 1.412841696s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
22972024-05-30T04:09:41.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 648.008811ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
22982024-05-30T04:09:41.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 1.111160708s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
22992024-05-30T04:09:41.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 321.773409ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
23002024-05-30T04:09:41.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 1.191343373s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
23012024-05-30T04:09:41.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 519.473458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
23022024-05-30T04:09:41.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 818.664964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
23032024-05-30T04:09:41.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 1.370765541s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
23042024-05-30T04:09:41.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 708.101964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
23052024-05-30T04:09:41.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 981.935525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
23062024-05-30T04:09:41.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 1.096592079s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
23072024-05-30T04:09:42.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 623.043731ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
23082024-05-30T04:09:42.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 1.32102673s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
23092024-05-30T04:09:42.312ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 721.635749ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
23102024-05-30T04:09:42.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 1.05685519s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
23112024-05-30T04:09:42.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 817.46484ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
23122024-05-30T04:09:42.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 861.935626ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
23132024-05-30T04:09:42.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 1.055279391s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
23142024-05-30T04:09:42.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 704.092285ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
23152024-05-30T04:09:42.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 1.108185608s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
23162024-05-30T04:09:42.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 1.453681446s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
23172024-05-30T04:09:42.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 898.830301ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
23182024-05-30T04:09:42.956ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 1.110851834s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
23192024-05-30T04:09:43.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 1.357961059s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
23202024-05-30T04:09:43.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 1.337739575s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
23212024-05-30T04:09:43.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 913.49333ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
23222024-05-30T04:09:43.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 1.130482903s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
23232024-05-30T04:09:43.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 898.537979ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
23242024-05-30T04:09:43.257ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService25"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::9"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte3"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::9]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.6:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23252024-05-30T04:09:43.258ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
23262024-05-30T04:09:43.280ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService27"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::8"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte4"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::8]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.5:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23272024-05-30T04:09:43.280ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
23282024-05-30T04:09:43.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 528.193317ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
23292024-05-30T04:09:43.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 669.50275ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
23302024-05-30T04:09:43.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 747.605234ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
23312024-05-30T04:09:43.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 849.799652ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
23322024-05-30T04:09:43.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 30.462365ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
23332024-05-30T04:09:43.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 55.95698ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
23342024-05-30T04:09:43.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 65.073338ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
23352024-05-30T04:09:43.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 822.545383ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
23362024-05-30T04:09:43.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 84.377906ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
23372024-05-30T04:09:43.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 232.76501ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
23382024-05-30T04:09:43.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 820.364324ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
23392024-05-30T04:09:43.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 652.914608ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
23402024-05-30T04:09:43.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 199.682156ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
23412024-05-30T04:09:44.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 779.468335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
23422024-05-30T04:09:44.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 262.953963ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
23432024-05-30T04:09:44.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 634.743071ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
23442024-05-30T04:09:44.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 596.547892ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
23452024-05-30T04:09:44.130ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46565
23462024-05-30T04:09:44.133ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 2595 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46565 req_id = 860e589c-3bc9-4668-9f49-0993c26aac09 response_code = 200 uri = /baseboard
23472024-05-30T04:09:44.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 764.692361ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
23482024-05-30T04:09:44.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 991.9501ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
23492024-05-30T04:09:44.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 728.201147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
23502024-05-30T04:09:44.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 953.272166ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
23512024-05-30T04:09:44.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 1.320351854s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
23522024-05-30T04:09:44.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 678.670181ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
23532024-05-30T04:09:44.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 570.354685ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
23542024-05-30T04:09:44.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 1.045554162s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
23552024-05-30T04:09:44.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 770.163574ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
23562024-05-30T04:09:44.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 1.135771584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
23572024-05-30T04:09:44.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 1.298907021s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
23582024-05-30T04:09:44.524ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService24"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte1"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
23592024-05-30T04:09:44.524ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
23602024-05-30T04:09:44.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 768.579641ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
23612024-05-30T04:09:44.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 1.302200348s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
23622024-05-30T04:09:44.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 1.045968643s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
23632024-05-30T04:09:44.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 677.87633ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
23642024-05-30T04:09:44.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 46.076772ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
23652024-05-30T04:09:44.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 1.142587522s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
23662024-05-30T04:09:44.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 122.492208ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
23672024-05-30T04:09:44.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 908.118633ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
23682024-05-30T04:09:44.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 1.395203354s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
23692024-05-30T04:09:44.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 1.146018901s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
23702024-05-30T04:09:44.951ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 5.75631181s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
23712024-05-30T04:09:44.972ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 258.370068ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
23722024-05-30T04:09:45.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 1.236574262s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
23732024-05-30T04:09:45.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 1.007145229s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
23742024-05-30T04:09:45.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 569.465418ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
23752024-05-30T04:09:45.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 404.94563ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
23762024-05-30T04:09:45.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 798.476805ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
23772024-05-30T04:09:45.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 1.050607342s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
23782024-05-30T04:09:45.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 1.464288561s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
23792024-05-30T04:09:45.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 1.152145571s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
23802024-05-30T04:09:45.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 505.401744ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
23812024-05-30T04:09:45.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 512.22153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
23822024-05-30T04:09:45.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 938.601899ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
23832024-05-30T04:09:45.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 1.437140528s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
23842024-05-30T04:09:45.794ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 639.3744ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
23852024-05-30T04:09:45.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 804.592651ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
23862024-05-30T04:09:45.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 1.250933305s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
23872024-05-30T04:09:45.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 633.350646ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
23882024-05-30T04:09:45.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 1.4018423s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
23892024-05-30T04:09:46.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 1.14091228s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
23902024-05-30T04:09:46.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 1.455196019s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
23912024-05-30T04:09:46.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 846.609678ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
23922024-05-30T04:09:46.184ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 775.55804ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
23932024-05-30T04:09:46.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 1.146638778s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
23942024-05-30T04:09:46.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 1.36879945s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
23952024-05-30T04:09:46.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 641.701427ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
23962024-05-30T04:09:46.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 671.370828ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
23972024-05-30T04:09:46.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 899.0721ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
23982024-05-30T04:09:46.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 593.218131ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
23992024-05-30T04:09:46.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 1.382798795s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
24002024-05-30T04:09:46.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 748.421541ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
24012024-05-30T04:09:46.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 963.179951ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
24022024-05-30T04:09:46.719ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService28"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte5"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
24032024-05-30T04:09:46.720ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24042024-05-30T04:09:46.813ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService26"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::c"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte2"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
24052024-05-30T04:09:46.813ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24062024-05-30T04:09:46.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 886.075504ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
24072024-05-30T04:09:46.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 521.343552ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
24082024-05-30T04:09:47.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 1.460398301s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
24092024-05-30T04:09:47.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 1.318746725s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
24102024-05-30T04:09:47.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 73.598896ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24112024-05-30T04:09:47.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 1.118769678s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
24122024-05-30T04:09:47.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 900.692785ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
24132024-05-30T04:09:47.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 80.307814ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24142024-05-30T04:09:47.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 1.41386001s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
24152024-05-30T04:09:47.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 1.290369975s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
24162024-05-30T04:09:47.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 28.655784ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24172024-05-30T04:09:47.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 534.074029ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
24182024-05-30T04:09:47.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 166.69843ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24192024-05-30T04:09:47.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 698.415681ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
24202024-05-30T04:09:47.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 1.16504985s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
24212024-05-30T04:09:47.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 62.144996ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24222024-05-30T04:09:47.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 915.184323ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
24232024-05-30T04:09:47.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 652.861053ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
24242024-05-30T04:09:47.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 149.135319ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24252024-05-30T04:09:47.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 1.286258537s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
24262024-05-30T04:09:47.540ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 209.97367ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24272024-05-30T04:09:47.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 1.030907128s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
24282024-05-30T04:09:47.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 983.671549ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
24292024-05-30T04:09:47.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 564.889054ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24302024-05-30T04:09:47.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 1.130265467s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
24312024-05-30T04:09:47.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 503.708396ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24322024-05-30T04:09:47.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 1.289466428s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
24332024-05-30T04:09:47.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 530.841919ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
24342024-05-30T04:09:48.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 1.181212326s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
24352024-05-30T04:09:48.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 997.245865ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
24362024-05-30T04:09:48.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 1.22752373s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
24372024-05-30T04:09:48.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 594.773382ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
24382024-05-30T04:09:48.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 637.830519ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24392024-05-30T04:09:48.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 734.957185ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
24402024-05-30T04:09:48.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 787.388137ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24412024-05-30T04:09:48.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 1.158096461s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
24422024-05-30T04:09:48.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 865.82541ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
24432024-05-30T04:09:48.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 689.58761ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
24442024-05-30T04:09:48.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 918.873858ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
24452024-05-30T04:09:48.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 1.199332491s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
24462024-05-30T04:09:48.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 524.612464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
24472024-05-30T04:09:48.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 1.025599712s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
24482024-05-30T04:09:48.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 1.13854945s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
24492024-05-30T04:09:48.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 1.354052651s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
24502024-05-30T04:09:48.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 840.815493ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
24512024-05-30T04:09:48.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 501.394133ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
24522024-05-30T04:09:48.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 1.361557867s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
24532024-05-30T04:09:48.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 1.013879846s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24542024-05-30T04:09:48.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 1.155591293s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
24552024-05-30T04:09:49.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 938.409753ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
24562024-05-30T04:09:49.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 635.646543ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24572024-05-30T04:09:49.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 766.703793ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
24582024-05-30T04:09:49.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 594.177477ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
24592024-05-30T04:09:49.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 1.189502564s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
24602024-05-30T04:09:49.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 943.614982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
24612024-05-30T04:09:49.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 615.212462ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
24622024-05-30T04:09:49.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 1.42026195s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
24632024-05-30T04:09:49.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 581.127713ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
24642024-05-30T04:09:49.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 989.743864ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
24652024-05-30T04:09:49.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 534.101052ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
24662024-05-30T04:09:49.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 1.104072407s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
24672024-05-30T04:09:49.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 619.481153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
24682024-05-30T04:09:49.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 918.856498ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24692024-05-30T04:09:49.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 916.980478ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
24702024-05-30T04:09:49.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed") failed: Property not found. retry in 1.422182157s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
24712024-05-30T04:09:49.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 874.500751ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
24722024-05-30T04:09:49.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 647.099935ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
24732024-05-30T04:09:49.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c") failed: Property not found. retry in 1.285795598s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
24742024-05-30T04:09:49.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 1.282194645s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24752024-05-30T04:09:49.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 869.440346ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
24762024-05-30T04:09:50.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 844.018272ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
24772024-05-30T04:09:50.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 1.469282744s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
24782024-05-30T04:09:50.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 599.371814ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
24792024-05-30T04:09:50.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 833.917512ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
24802024-05-30T04:09:50.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1") failed: Property not found. retry in 1.330294949s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
24812024-05-30T04:09:50.210ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 822.663084ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
24822024-05-30T04:09:50.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea") failed: Property not found. retry in 1.37834972s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
24832024-05-30T04:09:50.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab") failed: Property not found. retry in 1.193585855s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
24842024-05-30T04:09:50.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2") failed: Property not found. retry in 888.845831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
24852024-05-30T04:09:50.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f") failed: Property not found. retry in 1.451924547s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
24862024-05-30T04:09:50.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 1.349797717s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
24872024-05-30T04:09:50.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 1.109400104s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
24882024-05-30T04:09:50.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 1.317120031s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
24892024-05-30T04:09:50.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920") failed: Property not found. retry in 1.027023418s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
24902024-05-30T04:09:50.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 1.430435804s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
24912024-05-30T04:09:50.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232") failed: Property not found. retry in 922.983729ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
24922024-05-30T04:09:50.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 1.131068541s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
24932024-05-30T04:09:50.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128") failed: Property not found. retry in 1.321184035s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
24942024-05-30T04:09:51.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5") failed: Property not found. retry in 702.098585ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
24952024-05-30T04:09:51.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835") failed: Property not found. retry in 778.696502ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
24962024-05-30T04:09:51.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 646.823947ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
24972024-05-30T04:09:51.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144") failed: Property not found. retry in 718.497059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
24982024-05-30T04:09:51.815ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 900.055717ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
24992024-05-30T04:09:51.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 972.006972ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
25002024-05-30T04:09:51.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 1.33461601s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
25012024-05-30T04:09:52.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 916.175969ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
25022024-05-30T04:09:52.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 1.293641354s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
25032024-05-30T04:09:52.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 745.728542ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
25042024-05-30T04:09:52.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3") failed: Property not found. retry in 1.214216716s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
25052024-05-30T04:09:52.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 536.596522ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
25062024-05-30T04:09:52.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba") failed: Property not found. retry in 1.391290787s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
25072024-05-30T04:09:52.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f") failed: Property not found. retry in 1.085264114s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
25082024-05-30T04:09:53.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 543.648649ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
25092024-05-30T04:09:53.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 977.492767ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
25102024-05-30T04:09:53.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 620.681373ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
25112024-05-30T04:09:53.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4") failed: Property not found. retry in 1.425143519s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
25122024-05-30T04:09:54.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 1.414024213s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
25132024-05-30T04:09:54.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 580.56585ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
25142024-05-30T04:09:54.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 1.345960923s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
25152024-05-30T04:09:55.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 641.860237ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
25162024-05-30T04:09:56.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668") failed: Property not found. retry in 966.111019ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
25172024-05-30T04:09:56.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48") failed: Property not found. retry in 627.375251ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
25182024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_pantry_3ef0ace2-9c93-43ed-bb5d-824e6b6b0cab
25192024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_external_dns_6d9390f7-43dc-4e13-9714-d2a358e2f5d4
25202024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_nexus_5cc3fd7a-881a-45bf-9524-016003ecf668
25212024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_pantry_c2209168-b7ee-4e88-824f-9effc24609ea
25222024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_pantry_2bbcf07c-7614-4e39-9570-91a5c47d5232
25232024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_b5ac113d-4f55-4c3e-8718-a1f7414bba5f
25242024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_1f5dbf3d-e4c0-438a-82cb-4d43a6453128
25252024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_df8c46fb-39e3-4705-bde9-709842a66be5
25262024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_dbc6937e-eabc-4bd1-922e-b561820024c2
25272024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_15faa041-d273-4f44-b56e-98664dd6c144
25282024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_47b0a86a-e796-41a6-af75-fb0dfd4a86c1
25292024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_1f1da100-e042-4583-8561-6eff4c72a835
25302024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_3388152a-babe-4c26-b4bb-14448f905920
25312024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_crucible_dbfa33e5-1797-4930-9f90-903e48c8cf4c
25322024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_external_dns_56096d63-69cd-4c67-9c8f-fe757a4566d3
25332024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_nexus_c27f5e00-47bb-499a-ad9e-1146fabe9d48
25342024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_clickhouse_ca8c551e-b7e2-4add-a459-c77679e168ba
25352024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_nexus_d5da8fa4-2f8d-4c39-9334-8e19330e172f
25362024-05-30T04:09:57.168ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3054 zone = oxz_oximeter_8f3604d6-5231-4728-9097-caf093b736ed
25372024-05-30T04:09:57.169ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
25382024-05-30T04:09:57.170ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
25392024-05-30T04:09:57.172ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 24129020 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51321 req_id = ff98a91e-ddf8-48a3-900a-f0001bd93aff response_code = 204 uri = /omicron-zones
25402024-05-30T04:09:57.190ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1237
25412024-05-30T04:09:57.190ZINFOSledAgent (RSS): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
25422024-05-30T04:09:57.190ZINFOSledAgent (RSS): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
25432024-05-30T04:09:57.194ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:681
25442024-05-30T04:09:57.198ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:694
25452024-05-30T04:09:57.230ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV1 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV1 { addresses: [ V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:818
25462024-05-30T04:09:58.134ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25472024-05-30T04:09:58.246ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25482024-05-30T04:09:58.371ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25492024-05-30T04:09:58.460ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25502024-05-30T04:09:58.613ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25512024-05-30T04:09:58.677ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.132072036s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:417 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
25522024-05-30T04:09:58.856ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25532024-05-30T04:09:59.089ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25542024-05-30T04:09:59.356ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25552024-05-30T04:09:59.576ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25562024-05-30T04:10:00.195ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25572024-05-30T04:10:00.735ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25582024-05-30T04:10:01.574ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25592024-05-30T04:10:02.503ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25602024-05-30T04:10:02.856ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 16.367731676s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:417 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
25612024-05-30T04:10:03.037ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25622024-05-30T04:10:03.651ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25632024-05-30T04:10:05.420ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25642024-05-30T04:10:07.087ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4559c8be-8414-459c-b1cf-c6595c790cca/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:880
25652024-05-30T04:10:09.394ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "786b9bf7-8848-440f-8c20-02603c8df42e", "content-length": "173", "date": "Thu, 30 May 2024 04:10:09 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"dd908086-f372-434c-b66b-6e5770f567d8\"", request_id: "786b9bf7-8848-440f-8c20-02603c8df42e" } file = sled-agent/src/rack_setup/service.rs:880
25662024-05-30T04:10:12.978ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "28be8154-51d1-4d6b-a248-6a279f3fa0f2", "content-length": "173", "date": "Thu, 30 May 2024 04:10:12 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"dd908086-f372-434c-b66b-6e5770f567d8\"", request_id: "28be8154-51d1-4d6b-a248-6a279f3fa0f2" } file = sled-agent/src/rack_setup/service.rs:880
25672024-05-30T04:10:14.256ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34412
25682024-05-30T04:10:14.257ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 522 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34412 req_id = e0de5274-65a1-4112-8226-b69db507a011 response_code = 200 uri = /baseboard
25692024-05-30T04:10:17.181ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "f58852a7-9473-4dac-94d7-42c58b17a591", "content-length": "173", "date": "Thu, 30 May 2024 04:10:17 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"dd908086-f372-434c-b66b-6e5770f567d8\"", request_id: "f58852a7-9473-4dac-94d7-42c58b17a591" } file = sled-agent/src/rack_setup/service.rs:880
25702024-05-30T04:10:21.251ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "30a2f436-079f-4a85-800f-687944662ba5", "content-length": "173", "date": "Thu, 30 May 2024 04:10:20 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"dd908086-f372-434c-b66b-6e5770f567d8\"", request_id: "30a2f436-079f-4a85-800f-687944662ba5" } file = sled-agent/src/rack_setup/service.rs:880
25712024-05-30T04:10:26.417ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "214e7633-82db-4b17-bce9-cd6e646b3625", "content-length": "173", "date": "Thu, 30 May 2024 04:10:26 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"dd908086-f372-434c-b66b-6e5770f567d8\"", request_id: "214e7633-82db-4b17-bce9-cd6e646b3625" } file = sled-agent/src/rack_setup/service.rs:880
25722024-05-30T04:10:29.636ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
25732024-05-30T04:10:29.775ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59783
25742024-05-30T04:10:29.775ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25752024-05-30T04:10:29.862ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 86748 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59783 req_id = 153c0ab1-08f5-449a-a11d-098f0e4eee40 response_code = 200 uri = /inventory
25762024-05-30T04:10:29.863ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25772024-05-30T04:10:29.863ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
25782024-05-30T04:10:29.865ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
25792024-05-30T04:10:29.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 2501 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59783 req_id = 4998552e-597e-471c-b761-661bf8c1c178 response_code = 200 uri = /omicron-zones
25802024-05-30T04:10:34.000ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51253
25812024-05-30T04:10:34.002ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
25822024-05-30T04:10:34.003ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25832024-05-30T04:10:34.003ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25842024-05-30T04:10:34.003ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25852024-05-30T04:10:34.003ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
25862024-05-30T04:10:34.003ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
25872024-05-30T04:10:34.003ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
25882024-05-30T04:10:34.003ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 2395 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51253 req_id = a4cd9393-7136-436d-b4f2-26e4949fb929 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
25892024-05-30T04:10:34.005ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:890
25902024-05-30T04:10:34.251ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59151
25912024-05-30T04:10:34.251ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25922024-05-30T04:10:34.333ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 81850 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59151 req_id = d6ccefb3-7030-475d-ae03-685ce4d46111 response_code = 200 uri = /inventory
25932024-05-30T04:10:34.334ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25942024-05-30T04:10:34.334ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/config/omicron-zones.json file = common/src/ledger.rs:177
25952024-05-30T04:10:34.334ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/config/omicron-zones.json file = common/src/ledger.rs:177
25962024-05-30T04:10:34.335ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1014 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59151 req_id = 6cd392e6-5d15-4465-a68b-1e19391a6bf3 response_code = 200 uri = /omicron-zones
25972024-05-30T04:10:35.473ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35389
25982024-05-30T04:10:35.474ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
25992024-05-30T04:10:35.474ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26002024-05-30T04:10:35.474ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26012024-05-30T04:10:35.474ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26022024-05-30T04:10:35.474ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26032024-05-30T04:10:35.474ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26042024-05-30T04:10:35.474ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
26052024-05-30T04:10:35.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 357 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:35389 req_id = b1a77100-1796-47bb-b188-8f47494fd7ff response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26062024-05-30T04:10:35.585ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57593
26072024-05-30T04:10:35.585ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26082024-05-30T04:10:35.585ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26092024-05-30T04:10:35.585ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26102024-05-30T04:10:35.585ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26112024-05-30T04:10:35.585ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26122024-05-30T04:10:35.585ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26132024-05-30T04:10:35.585ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
26142024-05-30T04:10:35.585ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 270 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57593 req_id = 12c161a8-459c-4780-b42e-4fe86c6c76b5 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26152024-05-30T04:10:35.653ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33554
26162024-05-30T04:10:35.654ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 652 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33554 req_id = 4501ea64-f54f-48f8-9e51-45ca32bb4a74 response_code = 200 uri = /v2p
26172024-05-30T04:10:35.775ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64685
26182024-05-30T04:10:35.854ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 78670 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64685 req_id = 2f6e6501-7bce-4d23-bad6-0f3467938e82 response_code = 204 uri = /switch-ports
26192024-05-30T04:10:35.854ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60992
26202024-05-30T04:10:35.855ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
26212024-05-30T04:10:35.855ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 813 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64685 req_id = fbe78ffc-17a2-4233-958f-6a80cbe9dbc7 response_code = 200 uri = /network-bootstore-config
26222024-05-30T04:10:35.855ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26232024-05-30T04:10:35.855ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26242024-05-30T04:10:35.855ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26252024-05-30T04:10:35.855ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
26262024-05-30T04:10:35.855ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
26272024-05-30T04:10:35.855ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
26282024-05-30T04:10:35.855ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1016 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60992 req_id = 9bc70fa3-de6a-4d4b-a449-f90a6455b93e response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
26292024-05-30T04:10:35.879ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
26302024-05-30T04:10:35.879ZINFOSledAgent (bootstore): Writing ledger to /pool/int/75663f09-f516-40fc-852b-77e4a30319f5/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
26312024-05-30T04:10:35.880ZINFOSledAgent (bootstore): Writing ledger to /pool/int/804b9788-bd41-4d87-a91b-e568478fa644/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
26322024-05-30T04:10:35.881ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
26332024-05-30T04:10:35.881ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 2306 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64685 req_id = b0dbe8ef-45b0-42ac-8af6-4462af3c2c0b response_code = 204 uri = /network-bootstore-config
26342024-05-30T04:10:37.280ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37435
26352024-05-30T04:10:37.281ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 910 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37435 req_id = 04281d89-08a6-4a9c-a142-fe6915b3b697 response_code = 200 uri = /v2p
26362024-05-30T04:10:37.350ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65350
26372024-05-30T04:10:37.401ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 50967 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:65350 req_id = b7e1c24b-d4c7-4e3f-b5c9-892ceeab83ee response_code = 204 uri = /switch-ports
26382024-05-30T04:10:37.401ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41832
26392024-05-30T04:10:37.401ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 212 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41832 req_id = 6fbd8ce6-ceca-4a2f-99f6-546d03d73f87 response_code = 200 uri = /v2p
26402024-05-30T04:10:37.401ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 223 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65350 req_id = dc134fce-7f0e-4b17-a6c7-f1f3d6efe58e response_code = 200 uri = /network-bootstore-config
26412024-05-30T04:10:37.514ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61216
26422024-05-30T04:10:37.562ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46880 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61216 req_id = 37d6c157-6d31-455d-83a3-40b65b9e9c43 response_code = 204 uri = /switch-ports
26432024-05-30T04:10:37.562ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61216 req_id = a571f148-b963-4b07-b551-cb4965e5c868 response_code = 200 uri = /network-bootstore-config
26442024-05-30T04:10:44.396ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47847
26452024-05-30T04:10:44.397ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 549 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47847 req_id = 712407ac-43bd-4304-b1a5-3976884d6bce response_code = 200 uri = /baseboard
26462024-05-30T04:10:49.586ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:39972 remote_addr = [fd00:1122:3344:101::d]:33087 sled_id = dd908086-f372-434c-b66b-6e5770f567d8
26472024-05-30T04:10:49.591ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3929 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = 9aaee53c-3429-427f-973b-9146d23e6754 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
26482024-05-30T04:11:05.929ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60600
26492024-05-30T04:11:05.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 909 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60600 req_id = d94bc129-d334-4f74-ab7d-01ed30cf87d0 response_code = 200 uri = /v2p
26502024-05-30T04:11:06.211ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40935
26512024-05-30T04:11:06.259ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47789 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40935 req_id = 10b4fef2-3c68-4921-8649-2079662b545f response_code = 204 uri = /switch-ports
26522024-05-30T04:11:06.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40935 req_id = 0bb92e03-65ff-4dd1-9fe2-1f40c1b73a65 response_code = 200 uri = /network-bootstore-config
26532024-05-30T04:11:07.471ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36829
26542024-05-30T04:11:07.471ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36829 req_id = 1a4f7a7e-baa8-485c-9c39-4a546c42c161 response_code = 200 uri = /v2p
26552024-05-30T04:11:07.670ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54604
26562024-05-30T04:11:07.671ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 897 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54604 req_id = 522081dc-b3e5-4b69-956f-4588d09e6028 response_code = 200 uri = /v2p
26572024-05-30T04:11:07.766ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60959
26582024-05-30T04:11:07.810ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 43647 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60959 req_id = 956c783c-d66b-4c47-b4f6-792d93a6d20e response_code = 204 uri = /switch-ports
26592024-05-30T04:11:07.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 730 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60959 req_id = 9feade38-21a2-4e52-a986-cba42d3869d0 response_code = 200 uri = /network-bootstore-config
26602024-05-30T04:11:07.998ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47936
26612024-05-30T04:11:08.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 48280 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47936 req_id = 66e69e81-d4f3-4169-975f-c83657328b43 response_code = 204 uri = /switch-ports
26622024-05-30T04:11:08.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47936 req_id = c97e32a3-25c7-4bba-9621-d9724d64f0fc response_code = 200 uri = /network-bootstore-config
26632024-05-30T04:11:14.511ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46800
26642024-05-30T04:11:14.511ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 96 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46800 req_id = b84a9d6a-6c89-4e74-8f1e-41754d332e20 response_code = 200 uri = /baseboard
26652024-05-30T04:11:19.649ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1248 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = b6a8d081-708a-4512-8420-ec95132b5920 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
26662024-05-30T04:11:35.726ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46037
26672024-05-30T04:11:35.727ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 575 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46037 req_id = ceb8de37-4b1a-422e-8194-632d0e78b4fd response_code = 200 uri = /v2p
26682024-05-30T04:11:35.812ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42700
26692024-05-30T04:11:35.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 52978 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42700 req_id = a33810ab-d47f-4e3e-aa47-cd3438702ba7 response_code = 204 uri = /switch-ports
26702024-05-30T04:11:35.865ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42700 req_id = 1efd722d-605e-47e5-9875-dec6eb865af2 response_code = 200 uri = /network-bootstore-config
26712024-05-30T04:11:37.367ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56997
26722024-05-30T04:11:37.368ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 201 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56997 req_id = 4e9fd586-04c6-4d70-9ac4-3a7bdce2e008 response_code = 200 uri = /v2p
26732024-05-30T04:11:37.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42983
26742024-05-30T04:11:37.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42983 req_id = e79e9e94-d1cd-4122-83b0-5d85a604b798 response_code = 200 uri = /v2p
26752024-05-30T04:11:37.544ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44662
26762024-05-30T04:11:37.591ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47923 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44662 req_id = bc1d7bd1-7e08-416a-ad28-290ec7a0dd98 response_code = 204 uri = /switch-ports
26772024-05-30T04:11:37.592ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44662 req_id = 7fc1523c-7855-46f2-b553-12cf6f83f8c3 response_code = 200 uri = /network-bootstore-config
26782024-05-30T04:11:37.742ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60476
26792024-05-30T04:11:37.787ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44947 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60476 req_id = 2c006ca2-097e-4b46-80b4-4b098dab227d response_code = 204 uri = /switch-ports
26802024-05-30T04:11:37.788ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60476 req_id = 7a62bc85-f839-4884-80ff-5450a61b33f4 response_code = 200 uri = /network-bootstore-config
26812024-05-30T04:11:44.546ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:54293
26822024-05-30T04:11:44.546ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 118 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:54293 req_id = b811ae49-0624-4f40-bf3c-ed45b027241a response_code = 200 uri = /baseboard
26832024-05-30T04:11:49.648ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 538 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = deafcc0d-ea90-41ec-9d0e-d7b9e6e51d51 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
26842024-05-30T04:12:05.867ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50278
26852024-05-30T04:12:05.867ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50278 req_id = 80e411dd-3dc5-410e-9260-6b3fce0a65d1 response_code = 200 uri = /v2p
26862024-05-30T04:12:06.109ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53573
26872024-05-30T04:12:06.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46201 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:53573 req_id = 8299debb-41a8-46e2-ae98-0f923d5e2ae0 response_code = 204 uri = /switch-ports
26882024-05-30T04:12:06.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 693 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53573 req_id = 982ae0b2-7abd-4bfe-a6e6-7adc8b9360df response_code = 200 uri = /network-bootstore-config
26892024-05-30T04:12:07.372ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63886
26902024-05-30T04:12:07.372ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 170 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63886 req_id = 7b344942-c627-4b51-8b7f-22ad901f1cc4 response_code = 200 uri = /v2p
26912024-05-30T04:12:07.453ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46123
26922024-05-30T04:12:07.454ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 980 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46123 req_id = b641d173-d7c6-4408-903e-0d7fb9780f22 response_code = 200 uri = /v2p
26932024-05-30T04:12:07.713ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55340
26942024-05-30T04:12:07.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 49261 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:55340 req_id = bd426a90-5e50-4eeb-b332-e4b432a85b51 response_code = 204 uri = /switch-ports
26952024-05-30T04:12:07.762ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53078
26962024-05-30T04:12:07.810ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47768 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53078 req_id = 9fbfd5a3-7013-401f-8b4f-475c616a4ad1 response_code = 204 uri = /switch-ports
26972024-05-30T04:12:07.810ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55340 req_id = 8340f085-6b2f-436f-ac8b-e2c77fdf5c00 response_code = 200 uri = /network-bootstore-config
26982024-05-30T04:12:07.810ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53078 req_id = f99a2dfa-81c9-4c71-bd8e-046339cf7e09 response_code = 200 uri = /network-bootstore-config
26992024-05-30T04:12:14.578ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:57252
27002024-05-30T04:12:14.579ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 667 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:57252 req_id = 8dceccbc-701a-45fd-96dc-c5a5c307792a response_code = 200 uri = /baseboard
27012024-05-30T04:12:19.648ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 526 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = ba3445a5-06cf-4b97-b176-b19adca5b935 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
27022024-05-30T04:12:35.651ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56949
27032024-05-30T04:12:35.651ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 228 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56949 req_id = 162b81e7-b32d-4732-bdc6-58fa87165afc response_code = 200 uri = /v2p
27042024-05-30T04:12:35.824ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40654
27052024-05-30T04:12:35.871ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46995 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40654 req_id = 617e06ca-3301-4d34-9443-7ebe831b0a70 response_code = 204 uri = /switch-ports
27062024-05-30T04:12:35.871ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40654 req_id = 75dd3905-29b4-43b6-9cfe-8ffc4cd40004 response_code = 200 uri = /network-bootstore-config
27072024-05-30T04:12:37.379ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55174
27082024-05-30T04:12:37.379ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55174 req_id = 038de337-e8e9-409e-bc4b-f9b2b6576d1b response_code = 200 uri = /v2p
27092024-05-30T04:12:37.397ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41112
27102024-05-30T04:12:37.397ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41112 req_id = d36ef07a-fd03-48a1-b314-01bd63b38317 response_code = 200 uri = /v2p
27112024-05-30T04:12:37.471ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36160
27122024-05-30T04:12:37.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47913 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36160 req_id = ad0e43d0-62e4-4e58-8bb8-f744af120886 response_code = 204 uri = /switch-ports
27132024-05-30T04:12:37.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36160 req_id = f04a5ff5-ef9b-4afc-adec-76d9e212b618 response_code = 200 uri = /network-bootstore-config
27142024-05-30T04:12:37.526ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58680
27152024-05-30T04:12:37.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44879 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58680 req_id = fdcd7951-50fe-445c-8073-6db72dba650e response_code = 204 uri = /switch-ports
27162024-05-30T04:12:37.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58680 req_id = 026390a9-95cc-42cb-96b5-71d447c631e8 response_code = 200 uri = /network-bootstore-config
27172024-05-30T04:12:44.610ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:57651
27182024-05-30T04:12:44.610ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 125 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:57651 req_id = ff1d30ab-01da-4aa7-8cf3-a03a0d885b0e response_code = 200 uri = /baseboard
27192024-05-30T04:12:49.647ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 320 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = 8773db3f-1c4c-40ce-b256-cc21a4f2657f response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
27202024-05-30T04:13:01.906ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
27212024-05-30T04:13:05.897ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33069
27222024-05-30T04:13:05.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33069 req_id = 6fce6c28-c97c-4a1c-be4f-60cd44ef339f response_code = 200 uri = /v2p
27232024-05-30T04:13:06.300ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34855
27242024-05-30T04:13:06.347ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47032 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34855 req_id = 1386707b-cdcc-4721-b638-96e807e05540 response_code = 204 uri = /switch-ports
27252024-05-30T04:13:06.347ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34855 req_id = 5fa9c8f8-7001-46c0-9e38-4dda5153f3b7 response_code = 200 uri = /network-bootstore-config
27262024-05-30T04:13:07.494ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58603
27272024-05-30T04:13:07.495ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58603 req_id = 0bf68275-73e2-4814-bde9-ca1a798b5550 response_code = 200 uri = /v2p
27282024-05-30T04:13:07.581ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61604
27292024-05-30T04:13:07.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61604 req_id = ae10cb87-e405-4139-ad23-e9130b2b3730 response_code = 200 uri = /v2p
27302024-05-30T04:13:07.732ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39676
27312024-05-30T04:13:07.776ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44475 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:39676 req_id = 9f8f1a57-2b6b-4067-82d3-e82e387e4e1b response_code = 204 uri = /switch-ports
27322024-05-30T04:13:07.777ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35321
27332024-05-30T04:13:07.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3086 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39676 req_id = eacbcb9c-3403-43f0-b26d-e966e62c92d3 response_code = 200 uri = /network-bootstore-config
27342024-05-30T04:13:07.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44726 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35321 req_id = 0b5db3b3-2ae5-40ba-8a9f-a00c1abef569 response_code = 204 uri = /switch-ports
27352024-05-30T04:13:07.822ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35321 req_id = 84fcd351-e016-4825-a835-a9ab949ae106 response_code = 200 uri = /network-bootstore-config
27362024-05-30T04:13:14.642ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:63856
27372024-05-30T04:13:14.642ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 80 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:63856 req_id = 0430b149-ef65-476c-bdf4-8844da700ac8 response_code = 200 uri = /baseboard
27382024-05-30T04:13:19.694ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 535 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = 1d3af2ee-bf7f-4cac-8edc-b5becdaba5be response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
27392024-05-30T04:13:35.643ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38173
27402024-05-30T04:13:35.643ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 202 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38173 req_id = 0306c0a7-9cfa-4c41-a508-85bade4d987b response_code = 200 uri = /v2p
27412024-05-30T04:13:35.769ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64409
27422024-05-30T04:13:35.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45315 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64409 req_id = 24e4081a-585a-4fcc-b493-732d6354b6f7 response_code = 204 uri = /switch-ports
27432024-05-30T04:13:35.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64409 req_id = 630f4313-c441-4d08-86e3-a1ccbb8c10df response_code = 200 uri = /network-bootstore-config
27442024-05-30T04:13:37.462ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57401
27452024-05-30T04:13:37.462ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57401 req_id = dbe9f896-fe4f-464e-b553-36e6af9c06c2 response_code = 200 uri = /v2p
27462024-05-30T04:13:37.510ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62109
27472024-05-30T04:13:37.510ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62109 req_id = 43ff05eb-eb83-4830-a4ff-85360772c1b6 response_code = 200 uri = /v2p
27482024-05-30T04:13:37.565ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60088
27492024-05-30T04:13:37.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44559 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60088 req_id = e6be9dfd-3b8f-40d4-8a3f-3e5598642c44 response_code = 204 uri = /switch-ports
27502024-05-30T04:13:37.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 977 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60088 req_id = 6d4235f0-3340-4f90-8443-c9e33b691cd6 response_code = 200 uri = /network-bootstore-config
27512024-05-30T04:13:37.633ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36796
27522024-05-30T04:13:37.678ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44690 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36796 req_id = ade5e735-056c-4f3a-ba79-e49e622d44bb response_code = 204 uri = /switch-ports
27532024-05-30T04:13:37.678ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36796 req_id = a7de325a-9eb5-47f9-8fd3-1d87c6f26474 response_code = 200 uri = /network-bootstore-config
27542024-05-30T04:13:44.674ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:64208
27552024-05-30T04:13:44.674ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 64 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:64208 req_id = 1cd4163c-378b-4d9a-8a52-be22a6bfee75 response_code = 200 uri = /baseboard
27562024-05-30T04:13:49.692ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 539 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = 342cb8e1-be62-4de9-af03-6870cad5d439 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
27572024-05-30T04:14:05.896ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37045
27582024-05-30T04:14:05.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 978 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37045 req_id = 0f6562d1-6553-4ada-a2ea-f627bb6a1063 response_code = 200 uri = /v2p
27592024-05-30T04:14:06.056ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63423
27602024-05-30T04:14:06.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46215 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:63423 req_id = c0fad7c1-7b3e-4e71-8088-86f149fdeca3 response_code = 204 uri = /switch-ports
27612024-05-30T04:14:06.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63423 req_id = 862c901a-bcaa-44cb-b748-9c613dce377c response_code = 200 uri = /network-bootstore-config
27622024-05-30T04:14:07.382ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37480
27632024-05-30T04:14:07.382ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 182 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37480 req_id = c2a81112-46e6-4a04-ade9-e45301f6e319 response_code = 200 uri = /v2p
27642024-05-30T04:14:07.483ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33718
27652024-05-30T04:14:07.483ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33718 req_id = 2004852c-cb46-4838-af8b-a29a819d47bc response_code = 200 uri = /v2p
27662024-05-30T04:14:07.638ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47962
27672024-05-30T04:14:07.686ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47731 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47962 req_id = 5605d5c8-c6aa-4d9f-b4e2-21ba3bdd88db response_code = 204 uri = /switch-ports
27682024-05-30T04:14:07.686ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47962 req_id = 49e50be0-4b02-46cf-b057-340f8c889677 response_code = 200 uri = /network-bootstore-config
27692024-05-30T04:14:07.814ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37120
27702024-05-30T04:14:07.860ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46028 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37120 req_id = d8a8b997-a64d-4231-8e75-9db4562f7056 response_code = 204 uri = /switch-ports
27712024-05-30T04:14:07.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 735 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37120 req_id = 013973cb-750e-4fa8-845b-79688353ad21 response_code = 200 uri = /network-bootstore-config
27722024-05-30T04:14:14.708ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:51148
27732024-05-30T04:14:14.708ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 96 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:51148 req_id = c778023e-5dba-453f-a8d7-693ba1d353da response_code = 200 uri = /baseboard
27742024-05-30T04:14:19.692ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 586 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = 46f1a6dc-61ad-40fc-b007-048c1b7c2bf0 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
27752024-05-30T04:14:35.704ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44763
27762024-05-30T04:14:35.704ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 183 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44763 req_id = 19468a5e-7d16-4d0b-bc54-47fc1f4b261f response_code = 200 uri = /v2p
27772024-05-30T04:14:35.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54215
27782024-05-30T04:14:35.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 48396 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54215 req_id = 9162f624-4260-4cf2-b375-dec1b135aa8f response_code = 204 uri = /switch-ports
27792024-05-30T04:14:35.844ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54215 req_id = 383cfe3c-da6d-4d57-be12-6f87b1b290db response_code = 200 uri = /network-bootstore-config
27802024-05-30T04:14:37.421ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63773
27812024-05-30T04:14:37.421ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 746 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63773 req_id = a5ee7783-f9ef-42c9-8efe-e5c7da8ee083 response_code = 200 uri = /v2p
27822024-05-30T04:14:37.425ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56946
27832024-05-30T04:14:37.425ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56946 req_id = ce4d931d-c6d0-408a-86d2-aedadd25837c response_code = 200 uri = /v2p
27842024-05-30T04:14:37.510ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34395
27852024-05-30T04:14:37.559ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 49115 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34395 req_id = faaf7fe4-c387-4956-9954-b3eed1ea7ca0 response_code = 204 uri = /switch-ports
27862024-05-30T04:14:37.560ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34395 req_id = 756291cb-a32d-4dfa-a01e-43ed96732751 response_code = 200 uri = /network-bootstore-config
27872024-05-30T04:14:37.597ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59518
27882024-05-30T04:14:37.641ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44660 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59518 req_id = 0d849a2c-c302-41ac-92f1-92ed8b1ac9bd response_code = 204 uri = /switch-ports
27892024-05-30T04:14:37.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59518 req_id = 3757cb0a-36ea-49fd-9c3b-c998ff4f9ede response_code = 200 uri = /network-bootstore-config
27902024-05-30T04:14:44.743ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:60384
27912024-05-30T04:14:44.743ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 80 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:60384 req_id = 3fe728f4-cc90-4e8d-a7ca-9054632b6bdb response_code = 200 uri = /baseboard
27922024-05-30T04:14:49.692ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 573 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = ea787eec-d8ed-4ea7-8cc0-0d438d94a830 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
27932024-05-30T04:15:05.736ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35346
27942024-05-30T04:15:05.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 183 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35346 req_id = 41462c5b-7592-44f3-813e-1f154941e326 response_code = 200 uri = /v2p
27952024-05-30T04:15:06.015ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63658
27962024-05-30T04:15:06.016ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
27972024-05-30T04:15:06.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27982024-05-30T04:15:06.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27992024-05-30T04:15:06.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28002024-05-30T04:15:06.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28012024-05-30T04:15:06.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28022024-05-30T04:15:06.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
28032024-05-30T04:15:06.016ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 334 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63658 req_id = 10bc1873-f7e2-426b-9833-4ec04b2d9350 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
28042024-05-30T04:15:06.106ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44266
28052024-05-30T04:15:06.152ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45691 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44266 req_id = 2f3a3929-87f0-42f8-9f17-50bc27767aa2 response_code = 204 uri = /switch-ports
28062024-05-30T04:15:06.152ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 668 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44266 req_id = 611315e4-5cc2-4aee-82aa-caba96d2cb34 response_code = 200 uri = /network-bootstore-config
28072024-05-30T04:15:07.415ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39396
28082024-05-30T04:15:07.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 196 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39396 req_id = e4a85732-e5ae-4ab8-83fa-b43cee32f22a response_code = 200 uri = /v2p
28092024-05-30T04:15:07.460ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51612
28102024-05-30T04:15:07.460ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51612 req_id = d0d7b262-1bae-42ee-9aed-5a4f62114c28 response_code = 200 uri = /v2p
28112024-05-30T04:15:07.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60463
28122024-05-30T04:15:07.521ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
28132024-05-30T04:15:07.521ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28142024-05-30T04:15:07.522ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28152024-05-30T04:15:07.522ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28162024-05-30T04:15:07.522ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28172024-05-30T04:15:07.522ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28182024-05-30T04:15:07.522ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
28192024-05-30T04:15:07.522ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 355 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60463 req_id = 6aaf3d8e-1f8a-4a69-9ade-e55f66dcb52b response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
28202024-05-30T04:15:07.592ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63255
28212024-05-30T04:15:07.592ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e81480-505c-4272-9ace-9773d023170c, kind: Service { id: 56096d63-69cd-4c67-9c8f-fe757a4566d3 }, name: Name("external-dns-56096d63-69cd-4c67-9c8f-fe757a4566d3"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 13])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: f7c30c27-2106-484b-bc56-ac70941de2aa, kind: Service { id: 6d9390f7-43dc-4e13-9714-d2a358e2f5d4 }, name: Name("external-dns-6d9390f7-43dc-4e13-9714-d2a358e2f5d4"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 229, 173])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 22e4185f-95a2-4f41-bd9f-bdf79a1f811c, kind: Service { id: d5da8fa4-2f8d-4c39-9334-8e19330e172f }, name: Name("nexus-d5da8fa4-2f8d-4c39-9334-8e19330e172f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 213, 208])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 9d3b17eb-c104-4cd7-b817-286398ef2b2e, kind: Service { id: c27f5e00-47bb-499a-ad9e-1146fabe9d48 }, name: Name("nexus-c27f5e00-47bb-499a-ad9e-1146fabe9d48"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 244, 140])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: ad82ffd1-a739-4483-9b22-ad9556012366, kind: Service { id: 5cc3fd7a-881a-45bf-9524-016003ecf668 }, name: Name("nexus-5cc3fd7a-881a-45bf-9524-016003ecf668"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 105])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
28222024-05-30T04:15:07.592ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28232024-05-30T04:15:07.592ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28242024-05-30T04:15:07.592ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28252024-05-30T04:15:07.592ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
28262024-05-30T04:15:07.592ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
28272024-05-30T04:15:07.592ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
28282024-05-30T04:15:07.592ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 251 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63255 req_id = b8d82836-c28a-4622-9973-3dbd7462eb19 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
28292024-05-30T04:15:07.730ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52874
28302024-05-30T04:15:07.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45557 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52874 req_id = b15df732-7298-48ee-8259-dec8e84b572d response_code = 204 uri = /switch-ports
28312024-05-30T04:15:07.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52874 req_id = 3c4f85f6-5c37-4200-a905-0df7b66879d5 response_code = 200 uri = /network-bootstore-config
28322024-05-30T04:15:07.870ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48865
28332024-05-30T04:15:07.914ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44521 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48865 req_id = b11c6bf3-f284-4cd5-bcc3-3bef6a95b6e0 response_code = 204 uri = /switch-ports
28342024-05-30T04:15:07.914ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48865 req_id = a8c90f02-faac-489c-b4dc-28b02b19b693 response_code = 200 uri = /network-bootstore-config
28352024-05-30T04:15:14.776ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:63277
28362024-05-30T04:15:14.776ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 65 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:63277 req_id = 148820bf-592c-4079-957a-b6e3621825ad response_code = 200 uri = /baseboard
28372024-05-30T04:15:19.692ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 539 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = 27a31d5b-2021-4dee-b4f9-be74b9863366 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
28382024-05-30T04:15:35.663ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37833
28392024-05-30T04:15:35.664ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37833 req_id = 651a7f58-ecaa-46c5-8b74-9a71e1248ced response_code = 200 uri = /v2p
28402024-05-30T04:15:35.884ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46809
28412024-05-30T04:15:35.933ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 48939 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46809 req_id = e8f7592b-0707-4de8-af97-b1cefb60014a response_code = 204 uri = /switch-ports
28422024-05-30T04:15:35.933ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 102 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46809 req_id = d092ee25-5a1b-4e35-8417-209774d9fb7a response_code = 200 uri = /network-bootstore-config
28432024-05-30T04:15:37.337ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38042
28442024-05-30T04:15:37.337ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 200 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38042 req_id = d9db7095-3f46-466b-9ce9-23201406833d response_code = 200 uri = /v2p
28452024-05-30T04:15:37.475ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53234
28462024-05-30T04:15:37.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 49071 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53234 req_id = 5c444855-7fab-42ab-9334-4b6d82f3d1c8 response_code = 204 uri = /switch-ports
28472024-05-30T04:15:37.525ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:32960
28482024-05-30T04:15:37.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 216 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:32960 req_id = 4bd3c5d2-7701-4e1d-8bdb-d49ef78d5b61 response_code = 200 uri = /v2p
28492024-05-30T04:15:37.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 215 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53234 req_id = 8710d425-0537-4212-996b-badbe86fd6ba response_code = 200 uri = /network-bootstore-config
28502024-05-30T04:15:37.599ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60819
28512024-05-30T04:15:37.644ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44761 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60819 req_id = 29fb3132-8064-486c-9018-d302d7a55e76 response_code = 204 uri = /switch-ports
28522024-05-30T04:15:37.644ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60819 req_id = 8bae4cdf-f5a8-48e1-87c1-7dee7d037e42 response_code = 200 uri = /network-bootstore-config
28532024-05-30T04:15:44.808ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40781
28542024-05-30T04:15:44.808ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 87 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40781 req_id = d93285da-72da-4e22-bce0-42cbf43a2f45 response_code = 200 uri = /baseboard
28552024-05-30T04:15:49.769ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 492 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = 59ad9dd8-2e8e-4e98-8e63-3eb3b85b84b6 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
28562024-05-30T04:16:05.711ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65431
28572024-05-30T04:16:05.712ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65431 req_id = 963b2601-e7d9-4b0f-b270-4ed40a6ddaf3 response_code = 200 uri = /v2p
28582024-05-30T04:16:05.978ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44748
28592024-05-30T04:16:06.024ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46033 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:44748 req_id = 410cd0d7-3194-4aaf-a51b-5c4f1cdb415b response_code = 204 uri = /switch-ports
28602024-05-30T04:16:06.024ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44748 req_id = a0dadc33-64f7-4480-8b2d-c3332215e092 response_code = 200 uri = /network-bootstore-config
28612024-05-30T04:16:07.404ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64621
28622024-05-30T04:16:07.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64621 req_id = e1eae121-a01c-489f-bc4a-9c1b6df25ad6 response_code = 200 uri = /v2p
28632024-05-30T04:16:07.430ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61320
28642024-05-30T04:16:07.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61320 req_id = 6b3ab661-15da-4802-906c-4037c82908d0 response_code = 200 uri = /v2p
28652024-05-30T04:16:07.619ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60307
28662024-05-30T04:16:07.666ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46304 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60307 req_id = 103cd188-8c1b-4f90-981a-8429edfc29bf response_code = 204 uri = /switch-ports
28672024-05-30T04:16:07.666ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54133
28682024-05-30T04:16:07.669ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3257 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60307 req_id = aa17650e-53d2-4d7c-a356-bc2157a44c59 response_code = 200 uri = /network-bootstore-config
28692024-05-30T04:16:07.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 42589 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54133 req_id = e548660a-ff91-431b-9dce-0c3f8fbebe25 response_code = 204 uri = /switch-ports
28702024-05-30T04:16:07.709ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54133 req_id = 3fc57444-a00e-4e95-8c96-5d8837de2723 response_code = 200 uri = /network-bootstore-config
28712024-05-30T04:16:14.838ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36392
28722024-05-30T04:16:14.839ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 112 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36392 req_id = beb4d56f-8db4-45a6-b485-5ecaeee10db0 response_code = 200 uri = /baseboard
28732024-05-30T04:16:19.769ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 561 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = 696b4100-5a80-48c0-b895-29f7e639c2a9 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8
28742024-05-30T04:16:27.854ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57883
28752024-05-30T04:16:27.854ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57883 req_id = b1f5a4b3-a50e-444c-8981-1a61aa36bd53 response_code = 200 uri = /v2p
28762024-05-30T04:16:27.855ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:631 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(1275589) }
28772024-05-30T04:16:27.855ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 629 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57883 req_id = 4d3c6b34-7ecc-4864-a80a-3cea31581277 response_code = 204 uri = /v2p
28782024-05-30T04:16:28.017ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51723
28792024-05-30T04:16:28.021ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:546 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: fea462df-b254-4287-a947-a342cc88461f, kind: Instance { id: 3f458e42-852b-40df-ae34-e7c71d10bae3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(1275589), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: fea462df-b254-4287-a947-a342cc88461f, kind: Instance { id: 3f458e42-852b-40df-ae34-e7c71d10bae3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(1275589), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: fea462df-b254-4287-a947-a342cc88461f, kind: Instance { id: 3f458e42-852b-40df-ae34-e7c71d10bae3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(1275589), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(1275589))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: fea462df-b254-4287-a947-a342cc88461f, kind: Instance { id: 3f458e42-852b-40df-ae34-e7c71d10bae3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(1275589), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-53b30a23a56", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 78e1bdb7-6db9-4661-a894-5ed449ada38d, read_only_parent: Some(Volume { block_size: 512, id: 4f81eb9d-a7fd-4add-8460-f2dfd6f5e330, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 26c34ede-66b3-443b-8d4e-573690353318, key: Some("VGrFoM6ctTB5wVjzy7EXH1oMzOTJ7/e3OoGy3tAmO5M="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 78e1bdb7-6db9-4661-a894-5ed449ada38d, key: Some("ANvICRrbCFqVKPdepqMxy2h8Q+IARVsBojJy+qzXUUY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::18]:19003", "[fd00:1122:3344:101::13]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAADGDSK+WL5YAAANIr5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAxg0ivli+WAAADSK+WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzZjQ1OGU0Mi04NTJiLTQwZGYtYWUzNC1lN2M3MWQxMGJhZTMiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJT2FtamNNNjZCMDBZVnJLUnhGN2hYVzhjR1YzWTBqZm5GaElwcitnQkVBQyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 instance_runtime = InstanceRuntimeState { propolis_id: Some(fe68a66a-6283-428d-a0d8-78fc9912a771), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-30T04:16:27.659551102Z } metadata = InstanceMetadata { silo_id: c44c3a56-1c6c-45e4-84cb-09f661fdfc45, project_id: c9ef865c-a9ee-41b3-aab7-502ed3b6e928 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = fe68a66a-6283-428d-a0d8-78fc9912a771 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-05-30T04:16:27.631784Z }
28802024-05-30T04:16:28.021ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:580 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
28812024-05-30T04:16:28.021ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:938 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 propolis_id = fe68a66a-6283-428d-a0d8-78fc9912a771 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: fea462df-b254-4287-a947-a342cc88461f, kind: Instance { id: 3f458e42-852b-40df-ae34-e7c71d10bae3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(1275589), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: fea462df-b254-4287-a947-a342cc88461f, kind: Instance { id: 3f458e42-852b-40df-ae34-e7c71d10bae3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(1275589), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: fea462df-b254-4287-a947-a342cc88461f, kind: Instance { id: 3f458e42-852b-40df-ae34-e7c71d10bae3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(1275589), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(1275589))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: fea462df-b254-4287-a947-a342cc88461f, kind: Instance { id: 3f458e42-852b-40df-ae34-e7c71d10bae3 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(1275589), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-53b30a23a56", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 78e1bdb7-6db9-4661-a894-5ed449ada38d, read_only_parent: Some(Volume { block_size: 512, id: 4f81eb9d-a7fd-4add-8460-f2dfd6f5e330, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 26c34ede-66b3-443b-8d4e-573690353318, key: Some("VGrFoM6ctTB5wVjzy7EXH1oMzOTJ7/e3OoGy3tAmO5M="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 78e1bdb7-6db9-4661-a894-5ed449ada38d, key: Some("ANvICRrbCFqVKPdepqMxy2h8Q+IARVsBojJy+qzXUUY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::18]:19003", "[fd00:1122:3344:101::13]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAADGDSK+WL5YAAANIr5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAxg0ivli+WAAADSK+WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzZjQ1OGU0Mi04NTJiLTQwZGYtYWUzNC1lN2M3MWQxMGJhZTMiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJT2FtamNNNjZCMDBZVnJLUnhGN2hYVzhjR1YzWTBqZm5GaElwcitnQkVBQyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(fe68a66a-6283-428d-a0d8-78fc9912a771), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-30T04:16:27.659551102Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-05-30T04:16:27.631784Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
28822024-05-30T04:16:28.023ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 6289 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51723 req_id = a1938eba-1875-4ddf-bdf8-766d139dd173 response_code = 200 uri = /instances/3f458e42-852b-40df-ae34-e7c71d10bae3
28832024-05-30T04:16:28.133ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41143
28842024-05-30T04:16:28.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41143 req_id = 1df2cce1-a94d-408c-96bb-76ffce937c64 response_code = 200 uri = /instances/3f458e42-852b-40df-ae34-e7c71d10bae3/state
28852024-05-30T04:16:28.156ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 250, 13, 97]), slot: 0, vni: Vni { inner: 1275589 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 }, vnic: "vopte6" } }
28862024-05-30T04:16:28.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28872024-05-30T04:16:28.190ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771 file = illumos-utils/src/zone.rs:315 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
28882024-05-30T04:16:28.210ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771 file = illumos-utils/src/zone.rs:354 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
28892024-05-30T04:16:29.327ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1:0"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::b]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
28902024-05-30T04:16:29.327ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28912024-05-30T04:16:29.491ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 49.469362ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28922024-05-30T04:16:29.549ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 55.426653ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28932024-05-30T04:16:29.622ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 179.118494ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28942024-05-30T04:16:29.818ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 399.028039ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28952024-05-30T04:16:30.231ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 923.064496ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28962024-05-30T04:16:31.166ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 1.408708915s file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28972024-05-30T04:16:32.586ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 781.081074ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28982024-05-30T04:16:33.376ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 819.319639ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
28992024-05-30T04:16:34.207ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 777.090958ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 zone = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
29002024-05-30T04:16:35.012ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771 file = sled-agent/src/instance.rs:1423 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29012024-05-30T04:16:35.020ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771") failed: Property not found. retry in 51.322104ms file = illumos-utils/src/svc.rs:36 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29022024-05-30T04:16:35.082ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1432 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29032024-05-30T04:16:35.116ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1446 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29042024-05-30T04:16:35.117ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAADGDSK+WL5YAAANIr5YAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAxg0ivli+WAAADSK+WAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiIzZjQ1OGU0Mi04NTJiLTQwZGYtYWUzNC1lN2M3MWQxMGJhZTMiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJT2FtamNNNjZCMDBZVnJLUnhGN2hYVzhjR1YzWTBqZm5GaElwcitnQkVBQyJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-53b30a23a56", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 78e1bdb7-6db9-4661-a894-5ed449ada38d, read_only_parent: Some(Volume { block_size: 512, id: 4f81eb9d-a7fd-4add-8460-f2dfd6f5e330, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 26c34ede-66b3-443b-8d4e-573690353318, key: Some("VGrFoM6ctTB5wVjzy7EXH1oMzOTJ7/e3OoGy3tAmO5M="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 78e1bdb7-6db9-4661-a894-5ed449ada38d, key: Some("ANvICRrbCFqVKPdepqMxy2h8Q+IARVsBojJy+qzXUUY="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::18]:19003", "[fd00:1122:3344:101::13]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "vopte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 3f458e42-852b-40df-ae34-e7c71d10bae3, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: c9ef865c-a9ee-41b3-aab7-502ed3b6e928, silo_id: c44c3a56-1c6c-45e4-84cb-09f661fdfc45 }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:685 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29052024-05-30T04:16:35.238ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:687 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29062024-05-30T04:16:35.238ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-05-30T04:16:35.238709898Z } file = sled-agent/src/instance.rs:569 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29072024-05-30T04:16:35.238ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 new_instance_state = InstanceRuntimeState { propolis_id: Some(fe68a66a-6283-428d-a0d8-78fc9912a771), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-30T04:16:27.659551102Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-05-30T04:16:35.238709898Z } propolis_id = fe68a66a-6283-428d-a0d8-78fc9912a771
29082024-05-30T04:16:35.239ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(fe68a66a-6283-428d-a0d8-78fc9912a771), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-30T04:16:27.659551102Z }, propolis_id: fe68a66a-6283-428d-a0d8-78fc9912a771, vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-05-30T04:16:35.238709898Z } }
29092024-05-30T04:16:35.295ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-05-30T04:16:35.295656550Z } file = sled-agent/src/instance.rs:569 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29102024-05-30T04:16:35.295ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 new_instance_state = InstanceRuntimeState { propolis_id: Some(fe68a66a-6283-428d-a0d8-78fc9912a771), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-30T04:16:27.659551102Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-30T04:16:35.295656550Z } propolis_id = fe68a66a-6283-428d-a0d8-78fc9912a771
29112024-05-30T04:16:35.295ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(fe68a66a-6283-428d-a0d8-78fc9912a771), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-30T04:16:27.659551102Z }, propolis_id: fe68a66a-6283-428d-a0d8-78fc9912a771, vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-05-30T04:16:35.295656550Z } }
29122024-05-30T04:16:35.679ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61255
29132024-05-30T04:16:35.679ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61255 req_id = 626a483e-bc03-4c35-801e-7f136fa83cee response_code = 200 uri = /instances/3f458e42-852b-40df-ae34-e7c71d10bae3/state
29142024-05-30T04:16:35.735ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44329
29152024-05-30T04:16:35.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 778 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44329 req_id = 33ef6407-2292-41a0-a65a-1404fb6c15a1 response_code = 200 uri = /v2p
29162024-05-30T04:16:35.839ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51142
29172024-05-30T04:16:35.888ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 48279 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51142 req_id = 56b04e56-95e9-4973-9f20-d04d8c564006 response_code = 204 uri = /switch-ports
29182024-05-30T04:16:35.888ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51142 req_id = 252b3600-d0c6-445f-9804-b95e19398fbc response_code = 200 uri = /network-bootstore-config
29192024-05-30T04:16:37.400ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58456
29202024-05-30T04:16:37.400ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 176 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58456 req_id = 064eca28-f629-4d83-8937-6414d1750bf9 response_code = 200 uri = /instances/3f458e42-852b-40df-ae34-e7c71d10bae3/state
29212024-05-30T04:16:37.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60423
29222024-05-30T04:16:37.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60423 req_id = 8a02b341-2456-4f98-89cd-3725429ecc14 response_code = 200 uri = /instances/3f458e42-852b-40df-ae34-e7c71d10bae3/state
29232024-05-30T04:16:37.468ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51060
29242024-05-30T04:16:37.468ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51060 req_id = 9d9e3459-9ebb-4ac3-9c8f-1c0d350dc0f4 response_code = 200 uri = /v2p
29252024-05-30T04:16:37.520ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55431
29262024-05-30T04:16:37.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 732 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55431 req_id = 8bbefeac-3aa0-40df-903b-a33f3e414660 response_code = 200 uri = /v2p
29272024-05-30T04:16:37.595ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35464
29282024-05-30T04:16:37.641ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46678 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35464 req_id = dc409f8a-a94c-4e0c-b041-9a8322aa84ea response_code = 204 uri = /switch-ports
29292024-05-30T04:16:37.641ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36626
29302024-05-30T04:16:37.645ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3474 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35464 req_id = bb98aa4f-66bf-4985-953b-3d82f91c513b response_code = 200 uri = /network-bootstore-config
29312024-05-30T04:16:37.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45484 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:36626 req_id = 48697c33-c268-4753-a2e9-59e83491ebca response_code = 204 uri = /switch-ports
29322024-05-30T04:16:37.687ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36626 req_id = f26a59da-7215-448d-8db9-5e30c325b84a response_code = 200 uri = /network-bootstore-config
29332024-05-30T04:16:44.870ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:43547
29342024-05-30T04:16:44.871ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 612 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:43547 req_id = 73df1138-6d68-44d5-9d95-6f89df48066b response_code = 200 uri = /baseboard
29352024-05-30T04:16:45.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41653
29362024-05-30T04:16:45.187ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1266 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41653 req_id = d4cd457c-dd67-4e55-b64f-1f608dc1692b response_code = 200 uri = /instances/3f458e42-852b-40df-ae34-e7c71d10bae3/state
29372024-05-30T04:16:45.187ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-05-30T04:16:45.187546840Z } file = sled-agent/src/instance.rs:569 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29382024-05-30T04:16:45.187ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 new_instance_state = InstanceRuntimeState { propolis_id: Some(fe68a66a-6283-428d-a0d8-78fc9912a771), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-30T04:16:27.659551102Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-05-30T04:16:45.187546840Z } propolis_id = fe68a66a-6283-428d-a0d8-78fc9912a771
29392024-05-30T04:16:45.187ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(fe68a66a-6283-428d-a0d8-78fc9912a771), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-05-30T04:16:27.659551102Z }, propolis_id: fe68a66a-6283-428d-a0d8-78fc9912a771, vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-05-30T04:16:45.187546840Z } }
29402024-05-30T04:16:45.207ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-05-30T04:16:45.207036176Z } file = sled-agent/src/instance.rs:569 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29412024-05-30T04:16:45.207ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:591 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-30T04:16:45.207036176Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-05-30T04:16:45.207036176Z } propolis_id = fe68a66a-6283-428d-a0d8-78fc9912a771
29422024-05-30T04:16:45.207ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:606 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29432024-05-30T04:16:45.207ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29442024-05-30T04:16:45.208ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29452024-05-30T04:16:45.208ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/75663f09-f516-40fc-852b-77e4a30319f5/debug/bundle/zone", "/pool/int/804b9788-bd41-4d87-a91b-e568478fa644/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/3022fa0e-6566-4854-a1a9-1e0c5908d5b0/crypt/debug", "/pool/ext/1793c88e-7478-4741-8254-860fae35146b/crypt/debug", "/pool/ext/f61d361f-e4c0-48df-9191-ff705041ee76/crypt/debug", "/pool/ext/8ce3eaf9-2970-4cb9-a1f6-c0a6fa0594c7/crypt/debug", "/pool/ext/5d12239f-a80e-4f75-83d4-ebef24b6fea1/crypt/debug", "/pool/ext/a168fba6-15db-4743-814f-b76d3daf999f/crypt/debug", "/pool/ext/2efcf3a6-ef7f-48c2-8713-0dd19fe385d1/crypt/debug", "/pool/ext/dc70dd36-578e-4d7e-9ca2-11c36573efc3/crypt/debug", "/pool/ext/8f547a7a-6177-4ca0-82ce-6646cf05d8a6/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:437 zone_name = oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771
29462024-05-30T04:16:45.845ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1230 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771", bundle_id: 39b87f80-587e-43bb-9089-b147f407c824 }, time_created: 2024-05-30T04:16:45.208435660Z, version: 0, cause: TerminatedInstance }
29472024-05-30T04:16:45.846ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_fe68a66a-6283-428d-a0d8-78fc9912a771 file = sled-agent/src/instance.rs:780 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29482024-05-30T04:16:47.605ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:269 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29492024-05-30T04:16:47.630ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-30T04:16:45.207036176Z }, propolis_id: fe68a66a-6283-428d-a0d8-78fc9912a771, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-05-30T04:16:47.630813642Z } }
29502024-05-30T04:16:47.730ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60273
29512024-05-30T04:16:47.730ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60273 req_id = ba0c83dc-195e-49f3-aedc-f0e1876db14b response_code = 200 uri = /v2p
29522024-05-30T04:16:47.731ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:631 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(1275589) }
29532024-05-30T04:16:47.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:60273 req_id = 4a42fcdc-b2ec-4ae5-8b56-df619f36655c response_code = 204 uri = /v2p
29542024-05-30T04:16:47.896ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:491 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-05-30T04:16:45.207036176Z }, propolis_id: fe68a66a-6283-428d-a0d8-78fc9912a771, vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-05-30T04:16:47.630813642Z } }
29552024-05-30T04:16:47.899ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:723 instance_id = 3f458e42-852b-40df-ae34-e7c71d10bae3
29562024-05-30T04:16:48.093ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51396
29572024-05-30T04:16:48.093ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51396 req_id = 666ca6c0-2103-43cf-a4d9-094de155a98d response_code = 200 uri = /v2p
29582024-05-30T04:16:48.093ZINFOSledAgent (PortManager): Clearing mapping of virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:671 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 250, 13, 97])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(1275589) }
29592024-05-30T04:16:48.093ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:51396 req_id = 4f103250-5568-4789-b088-561f213b98ba response_code = 204 uri = /v2p
29602024-05-30T04:16:49.769ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 892 local_addr = [fd00:1122:3344:101::1]:39972 method = GET remote_addr = [fd00:1122:3344:101::d]:33087 req_id = dc16fd0e-28a5-4bea-bb62-bae9be123512 response_code = 200 sled_id = dd908086-f372-434c-b66b-6e5770f567d8 uri = /dd908086-f372-434c-b66b-6e5770f567d8