Max level shown:
1 [ Aug 5 16:42:32 Enabled. ]
2 [ Aug 5 16:42:32 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Aug 5 16:42:32 Method "start" exited with status 0. ]
4 [ Aug 5 16:42:32 Rereading configuration. ]
5 [ Aug 5 16:42:32 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-08-05T16:42:32.209ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-08-05T16:42:32.209ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-08-05T16:42:32.262ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-08-05T16:42:32.262ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-08-05T16:42:32.376ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-08-05T16:42:32.385ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-08-05T16:42:32.393ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-08-05T16:42:32.991ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-08-05T16:42:33.132ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:125
162024-08-05T16:42:33.132ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:137
172024-08-05T16:42:33.132ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:150
182024-08-05T16:42:33.132ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
192024-08-05T16:42:33.132ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:172 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
202024-08-05T16:42:33.132ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-08-05T16:42:33.135ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-08-05T16:42:33.135ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-08-05T16:42:33.135ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:186
242024-08-05T16:42:33.135ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-08-05T16:42:33.135ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_0.vdev
262024-08-05T16:42:33.135ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-08-05T16:42:33.136ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
282024-08-05T16:42:33.136ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-08-05T16:42:33.136ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-08-05T16:42:33.147ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-08-05T16:42:33.147ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool)
322024-08-05T16:42:33.266ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal }
332024-08-05T16:42:33.502ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:405
342024-08-05T16:42:33.502ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/m2_1.vdev
352024-08-05T16:42:33.502ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-08-05T16:42:33.502ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-08-05T16:42:33.502ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
382024-08-05T16:42:33.502ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-08-05T16:42:33.502ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-08-05T16:42:33.511ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-08-05T16:42:33.511ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool)
422024-08-05T16:42:33.620ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal }
432024-08-05T16:42:33.858ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_0.vdev
442024-08-05T16:42:33.858ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-08-05T16:42:33.858ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
482024-08-05T16:42:33.858ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-08-05T16:42:33.858ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_1.vdev
502024-08-05T16:42:33.858ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-08-05T16:42:33.858ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
522024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
532024-08-05T16:42:33.858ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
542024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-08-05T16:42:33.858ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_2.vdev
562024-08-05T16:42:33.858ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582024-08-05T16:42:33.858ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
592024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
602024-08-05T16:42:33.858ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-08-05T16:42:33.858ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_3.vdev
622024-08-05T16:42:33.858ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-08-05T16:42:33.858ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
642024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
652024-08-05T16:42:33.858ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
662024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-08-05T16:42:33.858ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_4.vdev
682024-08-05T16:42:33.858ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-08-05T16:42:33.858ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
722024-08-05T16:42:33.858ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-08-05T16:42:33.858ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_5.vdev
742024-08-05T16:42:33.858ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-08-05T16:42:33.858ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-08-05T16:42:33.858ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
782024-08-05T16:42:33.858ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-08-05T16:42:33.858ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_6.vdev
802024-08-05T16:42:33.858ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
812024-08-05T16:42:33.858ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
822024-08-05T16:42:33.859ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832024-08-05T16:42:33.859ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-08-05T16:42:33.859ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-08-05T16:42:33.859ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_7.vdev
862024-08-05T16:42:33.859ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
872024-08-05T16:42:33.859ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
882024-08-05T16:42:33.859ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
892024-08-05T16:42:33.859ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
902024-08-05T16:42:33.859ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
912024-08-05T16:42:33.859ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:242 vdev = /scratch/u2_8.vdev
922024-08-05T16:42:33.859ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:405
932024-08-05T16:42:33.859ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
942024-08-05T16:42:33.859ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
952024-08-05T16:42:33.859ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-08-05T16:42:33.859ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-08-05T16:42:33.859ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:96
982024-08-05T16:42:33.859ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:98
992024-08-05T16:42:33.859ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-08-05T16:42:33.859ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-08-05T16:42:33.859ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-08-05T16:42:33.859ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1032024-08-05T16:42:33.859ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:210
1042024-08-05T16:42:33.859ZINFOSledAgent (bootstore): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-08-05T16:42:33.859ZINFOSledAgent (bootstore): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-08-05T16:42:33.859ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-08-05T16:42:33.859ZINFOSledAgent (bootstore): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-08-05T16:42:33.859ZINFOSledAgent (bootstore): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-08-05T16:42:33.859ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-08-05T16:42:33.859ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:215
1112024-08-05T16:42:33.859ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:230
1122024-08-05T16:42:33.870ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:725
1132024-08-05T16:42:33.870ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 11490, tv_nsec: 811279875 } time_to_next_cleanup = 599.999915219s
1142024-08-05T16:42:33.870ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3664
1152024-08-05T16:42:33.870ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1162024-08-05T16:42:33.870ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-08-05T16:42:33.872ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-08-05T16:42:33.873ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-08-05T16:42:33.873ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-08-05T16:42:33.873ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1017
1212024-08-05T16:42:33.873ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1222024-08-05T16:42:33.873ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1232024-08-05T16:42:33.873ZINFOSledAgent (RSS): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-08-05T16:42:33.873ZINFOSledAgent (RSS): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-08-05T16:42:33.873ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1077
1262024-08-05T16:42:33.873ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1272024-08-05T16:42:33.873ZINFOSledAgent (RSS): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-08-05T16:42:33.873ZINFOSledAgent (RSS): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-08-05T16:42:33.873ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1118
1302024-08-05T16:42:33.873ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-08-05T16:42:33.873ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-08-05T16:42:33.873ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1332024-08-05T16:42:33.873ZINFOSledAgent (RSS): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-08-05T16:42:33.874ZINFOSledAgent (RSS): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-08-05T16:42:33.874ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-08-05T16:42:33.874ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1151
1372024-08-05T16:42:33.874ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-08-05T16:42:33.874ZINFOSledAgent (bootstore): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-08-05T16:42:33.874ZINFOSledAgent (bootstore): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-08-05T16:42:33.874ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-08-05T16:42:33.874ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 3abf3a4f-8820-40af-a01f-4369b42a5529, rack_id: b682b2a4-3c99-42b2-9e14-9bba53281f21, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-08-05T16:42:33.875ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:58920
1432024-08-05T16:42:33.875ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 3abf3a4f-8820-40af-a01f-4369b42a5529, rack_id: b682b2a4-3c99-42b2-9e14-9bba53281f21, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1442024-08-05T16:42:33.875ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1452024-08-05T16:42:33.875ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1462024-08-05T16:42:33.875ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:405
1472024-08-05T16:42:33.875ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:573
1482024-08-05T16:42:33.875ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1492024-08-05T16:42:33.875ZINFOSledAgent (StorageManager): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1502024-08-05T16:42:33.875ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1512024-08-05T16:42:33.875ZINFOSledAgent (StorageManager): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1522024-08-05T16:42:33.875ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:562
1532024-08-05T16:42:33.875ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:590
1542024-08-05T16:42:33.875ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:394
1552024-08-05T16:42:33.875ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
1562024-08-05T16:42:33.875ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:480 result = DisksManagementResult { status: [] }
1572024-08-05T16:42:33.877ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3898
1582024-08-05T16:42:33.877ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4271
1592024-08-05T16:42:33.907ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1602024-08-05T16:42:33.928ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1612024-08-05T16:42:33.946ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1622024-08-05T16:42:33.987ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:383 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1632024-08-05T16:42:33.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
1642024-08-05T16:42:33.987ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:395 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1652024-08-05T16:42:33.987ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1662024-08-05T16:42:33.990ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 59.979987ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "008a4174-5d87-469f-988e-cf2880a3cae7", "content-length": "124", "date": "Mon, 05 Aug 2024 16:42:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "008a4174-5d87-469f-988e-cf2880a3cae7" } file = clients/ddm-admin-client/src/lib.rs:177
1672024-08-05T16:42:34.000ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_4576fedf-479b-43b3-8fad-bb8b7ec274f8/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1682024-08-05T16:42:34.028ZINFOSledAgent: successfully created encrypted zvol: name="oxi_4576fedf-479b-43b3-8fad-bb8b7ec274f8/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1692024-08-05T16:42:34.028ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_4576fedf-479b-43b3-8fad-bb8b7ec274f8/swap" file = sled-agent/src/swap_device.rs:96
1702024-08-05T16:42:34.034ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:410 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1712024-08-05T16:42:34.034ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1722024-08-05T16:42:34.034ZINFOSledAgent (BackingFs): Ensuring dataset oxi_4576fedf-479b-43b3-8fad-bb8b7ec274f8/backing/fmd file = sled-agent/src/backing_fs.rs:136
1732024-08-05T16:42:34.052ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-08-05T16:42:34.052ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 117.429553ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8c8efff4-fb87-4159-b342-5e1e6fac9a11", "content-length": "124", "date": "Mon, 05 Aug 2024 16:42:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8c8efff4-fb87-4159-b342-5e1e6fac9a11" } file = clients/ddm-admin-client/src/lib.rs:177
1752024-08-05T16:42:34.125ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1762024-08-05T16:42:34.171ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-08-05T16:42:34.171ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 158.150594ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c80d7a6c-e81f-4ed7-9599-a3531481ffa4", "content-length": "124", "date": "Mon, 05 Aug 2024 16:42:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c80d7a6c-e81f-4ed7-9599-a3531481ffa4" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-08-05T16:42:34.331ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-08-05T16:42:34.331ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 233.055129ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "14002942-0f67-408b-a7c1-b228d309951a", "content-length": "124", "date": "Mon, 05 Aug 2024 16:42:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "14002942-0f67-408b-a7c1-b228d309951a" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-08-05T16:42:34.344ZINFOSledAgent (BackingFs): Mounting oxi_4576fedf-479b-43b3-8fad-bb8b7ec274f8/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1812024-08-05T16:42:34.356ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1822024-08-05T16:42:34.356ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1832024-08-05T16:42:34.356ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1842024-08-05T16:42:34.356ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1852024-08-05T16:42:34.567ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-08-05T16:42:34.568ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 114.925684ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f3c4d3d9-d8cb-44d1-b03c-96e11f37401b", "content-length": "124", "date": "Mon, 05 Aug 2024 16:42:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f3c4d3d9-d8cb-44d1-b03c-96e11f37401b" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-08-05T16:42:34.685ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-08-05T16:42:34.685ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 325.595018ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1808b149-2a7b-44b6-9640-7b8d9a9d9917", "content-length": "124", "date": "Mon, 05 Aug 2024 16:42:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1808b149-2a7b-44b6-9640-7b8d9a9d9917" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-08-05T16:42:35.013ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1902024-08-05T16:42:35.013ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 255.481837ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ca680fc1-143e-451a-ab72-89ae42791310", "content-length": "124", "date": "Mon, 05 Aug 2024 16:42:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ca680fc1-143e-451a-ab72-89ae42791310" } file = clients/ddm-admin-client/src/lib.rs:177
1912024-08-05T16:42:35.090ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1922024-08-05T16:42:35.090ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:40509 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1932024-08-05T16:42:35.090ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1942024-08-05T16:42:35.091ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:40509 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1952024-08-05T16:42:35.097ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 139.523994ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1962024-08-05T16:42:35.139ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1972024-08-05T16:42:35.139ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
1982024-08-05T16:42:35.139ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:908 underlay_address = fd00:1122:3344:101::1
1992024-08-05T16:42:35.139ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2002024-08-05T16:42:35.140ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2012024-08-05T16:42:35.140ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2022024-08-05T16:42:35.140ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2032024-08-05T16:42:35.140ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2042024-08-05T16:42:35.140ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2052024-08-05T16:42:35.140ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2062024-08-05T16:42:35.140ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2072024-08-05T16:42:35.141ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3664
2082024-08-05T16:42:35.141ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:58920
2092024-08-05T16:42:35.141ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2102024-08-05T16:42:35.141ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2112024-08-05T16:42:35.317ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2122024-08-05T16:42:35.317ZINFOSledAgent (RSS): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2132024-08-05T16:42:35.317ZINFOSledAgent (RSS): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2142024-08-05T16:42:35.317ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2152024-08-05T16:42:35.317ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2162024-08-05T16:42:35.317ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2172024-08-05T16:42:35.318ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 670.313838ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2182024-08-05T16:42:35.322ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2192024-08-05T16:42:35.356ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52434
2202024-08-05T16:42:35.356ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2212024-08-05T16:42:35.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52434 req_id = a1b37d09-e5a2-43a2-93f2-f39db13445d4 response_code = 200 uri = /inventory
2222024-08-05T16:42:35.386ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57302
2232024-08-05T16:42:35.386ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57302 req_id = 3d99c969-1953-438a-92f8-5a34d4319992 response_code = 200 uri = /sled-role
2242024-08-05T16:42:35.387ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
2252024-08-05T16:42:35.387ZINFOSledAgent (RSS): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2262024-08-05T16:42:35.387ZINFOSledAgent (RSS): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2272024-08-05T16:42:35.387ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:830
2282024-08-05T16:42:35.419ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: a3ae8142-d872-46ed-a8f5-1d52643d5715, pool_id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 5c841a52-6724-4851-93d0-4b1d5629a1af, pool_id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 5fdef7c1-4728-486f-b1af-649d5f2d2aa9, pool_id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 42e8892c-087f-4794-83a0-c0fbf49a30cc, pool_id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 2456e0c2-5c6b-44b7-aebc-71a4d8c49e81, pool_id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: ea8136fb-9e0e-47ee-be22-039a7aa90f13, pool_id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 5c671b95-d7f1-40ca-a86f-acf1e606bf26, pool_id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: dda42a79-a396-4af3-8e55-625341dd0cd5, pool_id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: ec5a428d-5f04-4e94-a324-dd976be4c40e, pool_id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:355 sled_address = [fd00:1122:3344:101::1]:12345
2292024-08-05T16:42:35.419ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43904
2302024-08-05T16:42:35.419ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:842 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2312024-08-05T16:42:35.419ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: a3ae8142-d872-46ed-a8f5-1d52643d5715, pool_id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 5c841a52-6724-4851-93d0-4b1d5629a1af, pool_id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 5fdef7c1-4728-486f-b1af-649d5f2d2aa9, pool_id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 42e8892c-087f-4794-83a0-c0fbf49a30cc, pool_id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 2456e0c2-5c6b-44b7-aebc-71a4d8c49e81, pool_id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: ea8136fb-9e0e-47ee-be22-039a7aa90f13, pool_id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 5c671b95-d7f1-40ca-a86f-acf1e606bf26, pool_id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: dda42a79-a396-4af3-8e55-625341dd0cd5, pool_id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: ec5a428d-5f04-4e94-a324-dd976be4c40e, pool_id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:405
2322024-08-05T16:42:35.419ZINFOSledAgent (StorageManager): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2332024-08-05T16:42:35.419ZINFOSledAgent (StorageManager): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2342024-08-05T16:42:35.419ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:644 request = omicron_physical_disks_ensure
2352024-08-05T16:42:35.419ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:374
2362024-08-05T16:42:35.419ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:414
2372024-08-05T16:42:35.419ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2382024-08-05T16:42:35.419ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2392024-08-05T16:42:35.541ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2402024-08-05T16:42:35.541ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool)
2412024-08-05T16:42:36.148ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 668.379393ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2422024-08-05T16:42:36.149ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2432024-08-05T16:42:36.498ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }
2442024-08-05T16:42:36.660ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:348
2452024-08-05T16:42:36.660ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:350
2462024-08-05T16:42:36.660ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:354
2472024-08-05T16:42:36.660ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:356
2482024-08-05T16:42:36.660ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2492024-08-05T16:42:36.660ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2502024-08-05T16:42:36.660ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
2512024-08-05T16:42:36.841ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.065833161s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2522024-08-05T16:42:37.195ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2475
2532024-08-05T16:42:37.195ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2693
2542024-08-05T16:42:37.195ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2345
2552024-08-05T16:42:37.195ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2392
2562024-08-05T16:42:37.195ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2774
2572024-08-05T16:42:37.195ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2816
2582024-08-05T16:42:37.195ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2592024-08-05T16:42:37.196ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2602024-08-05T16:42:37.197ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2618
2612024-08-05T16:42:37.198ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2622024-08-05T16:42:37.380ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2386
2632024-08-05T16:42:37.381ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='acf91d2e-9ebb-402e-bf9a-466e661aef85'/> <propval type="astring" name="rack_id" value='b682b2a4-3c99-42b2-9e14-9bba53281f21'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe2a:2d7d'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='3abf3a4f-8820-40af-a01f-4369b42a5529'/> <propval type="astring" name="rack_id" value='b682b2a4-3c99-42b2-9e14-9bba53281f21'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='3abf3a4f-8820-40af-a01f-4369b42a5529'/> <propval type="astring" name="rack_uuid" value='b682b2a4-3c99-42b2-9e14-9bba53281f21'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='3abf3a4f-8820-40af-a01f-4369b42a5529'/> <propval type="astring" name="rack_uuid" value='b682b2a4-3c99-42b2-9e14-9bba53281f21'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2642024-08-05T16:42:37.381ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_switch
2652024-08-05T16:42:37.539ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_bd033efe-47fe-4ba2-8bc4-ba99e628ada7/crypt/zone file = sled-storage/src/dataset.rs:411
2662024-08-05T16:42:38.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 33.73814ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2672024-08-05T16:42:38.491ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2682024-08-05T16:42:38.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 68.42278ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2692024-08-05T16:42:39.083ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.414958127s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2702024-08-05T16:42:39.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 147.948151ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2712024-08-05T16:42:39.631ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2722024-08-05T16:42:39.631ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2732024-08-05T16:42:39.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 352.600336ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2742024-08-05T16:42:40.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 815.007348ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2752024-08-05T16:42:40.523ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_bd033efe-47fe-4ba2-8bc4-ba99e628ada7 file = sled-storage/src/dataset.rs:496
2762024-08-05T16:42:40.730ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2772024-08-05T16:42:40.736ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:512
2782024-08-05T16:42:40.736ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:429
2792024-08-05T16:42:40.736ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:414
2802024-08-05T16:42:40.736ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
2812024-08-05T16:42:40.736ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2822024-08-05T16:42:40.955ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2832024-08-05T16:42:40.955ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 1aa678c2-e32d-4428-a115-7e698777172b (zpool)
2842024-08-05T16:42:42.128ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }
2852024-08-05T16:42:42.361ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:348
2862024-08-05T16:42:42.361ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:350
2872024-08-05T16:42:42.361ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:354
2882024-08-05T16:42:42.361ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:356
2892024-08-05T16:42:42.361ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2902024-08-05T16:42:42.362ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2912024-08-05T16:42:42.362ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
2922024-08-05T16:42:43.093ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2932024-08-05T16:42:43.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 963.355659ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2942024-08-05T16:42:43.094ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2952024-08-05T16:42:43.356ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2962024-08-05T16:42:43.593ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 10.845809831s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2972024-08-05T16:42:43.600ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1aa678c2-e32d-4428-a115-7e698777172b/crypt/zone file = sled-storage/src/dataset.rs:411
2982024-08-05T16:42:44.760ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
2992024-08-05T16:42:45.210ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
3002024-08-05T16:42:45.211ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
3012024-08-05T16:42:45.213ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3922 new = [fd00:1122:3344:101::2, ::1] old = [::1]
3022024-08-05T16:42:45.213ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3938
3032024-08-05T16:42:45.213ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:518 zone = oxz_switch
3042024-08-05T16:42:45.311ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1aa678c2-e32d-4428-a115-7e698777172b file = sled-storage/src/dataset.rs:496
3052024-08-05T16:42:45.319ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:512
3062024-08-05T16:42:45.319ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:429
3072024-08-05T16:42:45.319ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:414
3082024-08-05T16:42:45.319ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3092024-08-05T16:42:45.319ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3102024-08-05T16:42:45.329ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3112024-08-05T16:42:45.329ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool)
3122024-08-05T16:42:45.439ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }
3132024-08-05T16:42:45.448ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:348
3142024-08-05T16:42:45.448ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:350
3152024-08-05T16:42:45.448ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:354
3162024-08-05T16:42:45.448ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:356
3172024-08-05T16:42:45.448ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3182024-08-05T16:42:45.448ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3192024-08-05T16:42:45.448ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3202024-08-05T16:42:45.490ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3212024-08-05T16:42:45.498ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396/crypt/zone file = sled-storage/src/dataset.rs:411
3222024-08-05T16:42:45.639ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 file = sled-storage/src/dataset.rs:496
3232024-08-05T16:42:45.648ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:512
3242024-08-05T16:42:45.648ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:429
3252024-08-05T16:42:45.648ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:414
3262024-08-05T16:42:45.648ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3272024-08-05T16:42:45.648ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3282024-08-05T16:42:45.658ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3292024-08-05T16:42:45.658ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool)
3302024-08-05T16:42:45.769ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }
3312024-08-05T16:42:45.777ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:348
3322024-08-05T16:42:45.777ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:350
3332024-08-05T16:42:45.777ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:354
3342024-08-05T16:42:45.777ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:356
3352024-08-05T16:42:45.777ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3362024-08-05T16:42:45.777ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3372024-08-05T16:42:45.777ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3382024-08-05T16:42:45.781ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
3392024-08-05T16:42:45.822ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3402024-08-05T16:42:45.830ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_939962e4-70b1-47be-b2d6-938c818b62c4/crypt/zone file = sled-storage/src/dataset.rs:411
3412024-08-05T16:42:45.975ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_939962e4-70b1-47be-b2d6-938c818b62c4 file = sled-storage/src/dataset.rs:496
3422024-08-05T16:42:45.983ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:512
3432024-08-05T16:42:45.983ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:429
3442024-08-05T16:42:45.983ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:414
3452024-08-05T16:42:45.983ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3462024-08-05T16:42:45.983ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3472024-08-05T16:42:45.993ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3482024-08-05T16:42:45.993ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool)
3492024-08-05T16:42:46.106ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }
3502024-08-05T16:42:46.115ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:348
3512024-08-05T16:42:46.115ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:350
3522024-08-05T16:42:46.115ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:354
3532024-08-05T16:42:46.115ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:356
3542024-08-05T16:42:46.115ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3552024-08-05T16:42:46.115ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3562024-08-05T16:42:46.115ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3572024-08-05T16:42:46.158ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3582024-08-05T16:42:46.166ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_52c33af8-e5fc-4cb6-93db-7a529f23be77/crypt/zone file = sled-storage/src/dataset.rs:411
3592024-08-05T16:42:46.308ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_52c33af8-e5fc-4cb6-93db-7a529f23be77 file = sled-storage/src/dataset.rs:496
3602024-08-05T16:42:46.316ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:512
3612024-08-05T16:42:46.316ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:429
3622024-08-05T16:42:46.316ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:414
3632024-08-05T16:42:46.316ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3642024-08-05T16:42:46.316ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3652024-08-05T16:42:46.326ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3662024-08-05T16:42:46.326ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool)
3672024-08-05T16:42:46.440ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }
3682024-08-05T16:42:46.448ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:348
3692024-08-05T16:42:46.448ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:350
3702024-08-05T16:42:46.448ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:354
3712024-08-05T16:42:46.448ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:356
3722024-08-05T16:42:46.448ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3732024-08-05T16:42:46.448ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3742024-08-05T16:42:46.448ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3752024-08-05T16:42:46.493ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3762024-08-05T16:42:46.501ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_cb14544a-cc03-4a04-84a7-da1d51f0a71d/crypt/zone file = sled-storage/src/dataset.rs:411
3772024-08-05T16:42:46.521ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
3782024-08-05T16:42:46.646ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_cb14544a-cc03-4a04-84a7-da1d51f0a71d file = sled-storage/src/dataset.rs:496
3792024-08-05T16:42:46.653ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:512
3802024-08-05T16:42:46.653ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:429
3812024-08-05T16:42:46.653ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:414
3822024-08-05T16:42:46.654ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
3832024-08-05T16:42:46.654ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3842024-08-05T16:42:46.663ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3852024-08-05T16:42:46.663ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool)
3862024-08-05T16:42:46.779ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }
3872024-08-05T16:42:46.787ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:348
3882024-08-05T16:42:46.787ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:350
3892024-08-05T16:42:46.787ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:354
3902024-08-05T16:42:46.787ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:356
3912024-08-05T16:42:46.787ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3922024-08-05T16:42:46.787ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3932024-08-05T16:42:46.787ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
3942024-08-05T16:42:46.791ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
3952024-08-05T16:42:46.828ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3962024-08-05T16:42:46.836ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2cb76428-77cf-4416-ab3e-f66fc49a03b4/crypt/zone file = sled-storage/src/dataset.rs:411
3972024-08-05T16:42:46.982ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2cb76428-77cf-4416-ab3e-f66fc49a03b4 file = sled-storage/src/dataset.rs:496
3982024-08-05T16:42:46.989ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:512
3992024-08-05T16:42:46.990ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:429
4002024-08-05T16:42:46.990ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:414
4012024-08-05T16:42:46.990ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
4022024-08-05T16:42:46.990ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4032024-08-05T16:42:47.000ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
4042024-08-05T16:42:47.000ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool)
4052024-08-05T16:42:47.110ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }
4062024-08-05T16:42:47.119ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:348
4072024-08-05T16:42:47.119ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:350
4082024-08-05T16:42:47.119ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:354
4092024-08-05T16:42:47.119ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:356
4102024-08-05T16:42:47.119ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4112024-08-05T16:42:47.119ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4122024-08-05T16:42:47.119ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
4132024-08-05T16:42:47.160ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4142024-08-05T16:42:47.168ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1a42fe8d-3436-46f3-8051-0d2ff2a0be4f/crypt/zone file = sled-storage/src/dataset.rs:411
4152024-08-05T16:42:47.317ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1a42fe8d-3436-46f3-8051-0d2ff2a0be4f file = sled-storage/src/dataset.rs:496
4162024-08-05T16:42:47.326ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:512
4172024-08-05T16:42:47.326ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:429
4182024-08-05T16:42:47.326ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:414
4192024-08-05T16:42:47.326ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:491
4202024-08-05T16:42:47.326ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4212024-08-05T16:42:47.336ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4222024-08-05T16:42:47.336ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool)
4232024-08-05T16:42:47.453ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:304 zpool = ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }
4242024-08-05T16:42:47.461ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:348
4252024-08-05T16:42:47.461ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:350
4262024-08-05T16:42:47.461ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:354
4272024-08-05T16:42:47.461ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:356
4282024-08-05T16:42:47.461ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4292024-08-05T16:42:47.461ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4302024-08-05T16:42:47.461ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:368
4312024-08-05T16:42:47.504ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4322024-08-05T16:42:47.512ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_85b774a7-53a2-4f22-a027-1f642ba2d7f9/crypt/zone file = sled-storage/src/dataset.rs:411
4332024-08-05T16:42:47.655ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_85b774a7-53a2-4f22-a027-1f642ba2d7f9 file = sled-storage/src/dataset.rs:496
4342024-08-05T16:42:47.663ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:512
4352024-08-05T16:42:47.663ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:429
4362024-08-05T16:42:47.663ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4372024-08-05T16:42:47.663ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4576fedf-479b-43b3-8fad-bb8b7ec274f8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: d09b3f64-0a36-4052-8233-a94e62e662e2 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4382024-08-05T16:42:47.663ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4392024-08-05T16:42:47.663ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4402024-08-05T16:42:47.664ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4412024-08-05T16:42:47.664ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:846 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4422024-08-05T16:42:47.664ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4432024-08-05T16:42:47.664ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:872 generation = Generation(1) sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4442024-08-05T16:42:47.664ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:877 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4452024-08-05T16:42:47.664ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:882 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4462024-08-05T16:42:47.664ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:116 gen = Generation(1) sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4472024-08-05T16:42:47.664ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:888 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4482024-08-05T16:42:47.664ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:838 gen = Generation(1)
4492024-08-05T16:42:47.664ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:893 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4502024-08-05T16:42:47.664ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 12245116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:43904 req_id = 44d6dcac-33c1-4c60-a33d-f052d3edf127 response_code = 200 uri = /omicron-physical-disks
4512024-08-05T16:42:47.695ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
4522024-08-05T16:42:47.695ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55797
4532024-08-05T16:42:47.695ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4542024-08-05T16:42:47.695ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4552024-08-05T16:42:47.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4562024-08-05T16:42:47.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
4572024-08-05T16:42:47.794ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4582024-08-05T16:42:47.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
4592024-08-05T16:42:47.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4602024-08-05T16:42:47.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4612024-08-05T16:42:47.879ZINFOSledAgent (ServiceManager): No ledger in /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:188
4622024-08-05T16:42:47.879ZINFOSledAgent (ServiceManager): No ledger in /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:188
4632024-08-05T16:42:47.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4642024-08-05T16:42:47.923ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4652024-08-05T16:42:47.923ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4662024-08-05T16:42:47.923ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
4672024-08-05T16:42:47.975ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875 file = illumos-utils/src/zone.rs:316
4682024-08-05T16:42:47.978ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56 file = illumos-utils/src/zone.rs:316
4692024-08-05T16:42:47.982ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19 file = illumos-utils/src/zone.rs:316
4702024-08-05T16:42:47.994ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875 file = illumos-utils/src/zone.rs:354
4712024-08-05T16:42:47.998ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56 file = illumos-utils/src/zone.rs:354
4722024-08-05T16:42:48.002ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19 file = illumos-utils/src/zone.rs:354
4732024-08-05T16:42:48.521ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4742024-08-05T16:42:48.797ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
4752024-08-05T16:42:49.473ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4762024-08-05T16:42:49.474ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4772024-08-05T16:42:49.950ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4782024-08-05T16:42:49.950ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
4792024-08-05T16:42:50.433ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4802024-08-05T16:42:50.433ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4812024-08-05T16:42:50.433ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4822024-08-05T16:42:50.433ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
4832024-08-05T16:42:50.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 50.548963ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
4842024-08-05T16:42:50.446ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
4852024-08-05T16:42:50.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 79.821833ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
4862024-08-05T16:42:50.566ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 55.827991ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
4872024-08-05T16:42:50.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 32.606347ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
4882024-08-05T16:42:50.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 120.092936ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
4892024-08-05T16:42:50.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 113.558994ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
4902024-08-05T16:42:50.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 133.6297ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
4912024-08-05T16:42:50.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 279.161549ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
4922024-08-05T16:42:50.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 270.398765ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
4932024-08-05T16:42:50.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 144.53606ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
4942024-08-05T16:42:51.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 427.126483ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
4952024-08-05T16:42:51.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 229.582987ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
4962024-08-05T16:42:51.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 241.053817ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
4972024-08-05T16:42:51.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 709.904483ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
4982024-08-05T16:42:51.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 736.942138ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
4992024-08-05T16:42:51.520ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 971.009818ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
5002024-08-05T16:42:52.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 966.063975ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
5012024-08-05T16:42:52.177ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 781.110945ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
5022024-08-05T16:42:52.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 734.470834ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
5032024-08-05T16:42:52.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 743.757803ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
5042024-08-05T16:42:53.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 893.599751ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
5052024-08-05T16:42:53.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 1.46543435s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
5062024-08-05T16:42:53.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 794.176725ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
5072024-08-05T16:42:53.819ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3946
5082024-08-05T16:42:53.819ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:3956 gateway = fd00:1122:3344:101::1
5092024-08-05T16:42:53.825ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4035
5102024-08-05T16:42:53.953ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4083
5112024-08-05T16:42:53.953ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4110
5122024-08-05T16:42:53.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 1.206315818s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
5132024-08-05T16:42:53.997ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4123
5142024-08-05T16:42:53.997ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:3987
5152024-08-05T16:42:54.054ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4029
5162024-08-05T16:42:54.055ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4090 rack_subnet = fd00:1122:3344::
5172024-08-05T16:42:54.074ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4101
5182024-08-05T16:42:54.074ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4143
5192024-08-05T16:42:54.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:52596
5202024-08-05T16:42:54.101ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:52596 req_id = 5c1164d3-2727-41e2-9a82-cc505746da64 response_code = 200 uri = /sled-identifiers
5212024-08-05T16:42:54.162ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4175
5222024-08-05T16:42:54.162ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4181
5232024-08-05T16:42:54.271ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4222
5242024-08-05T16:42:54.271ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5252024-08-05T16:42:54.303ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5262024-08-05T16:42:54.335ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:675
5272024-08-05T16:42:54.335ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:713
5282024-08-05T16:42:54.336ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:716 uptime = 0
5292024-08-05T16:42:54.336ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5302024-08-05T16:42:54.371ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
5312024-08-05T16:42:54.391ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
5322024-08-05T16:42:54.408ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:405
5332024-08-05T16:42:54.408ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5342024-08-05T16:42:54.408ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5352024-08-05T16:42:54.408ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5362024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5372024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5382024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5392024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5402024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5412024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5422024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5432024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5442024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5452024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5462024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5472024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5482024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5492024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5502024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5512024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5522024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5532024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:601 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5542024-08-05T16:42:54.409ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:622 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5552024-08-05T16:42:54.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875") failed: Property not found. retry in 1.347197649s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
5562024-08-05T16:42:54.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56") failed: Property not found. retry in 903.3388ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
5572024-08-05T16:42:55.171ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19") failed: Property not found. retry in 1.241292377s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
5582024-08-05T16:42:56.429ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_internal_dns_fe17df80-010a-4b68-9ebc-6a94437be875
5592024-08-05T16:42:56.429ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_internal_dns_83a194e2-3464-4a84-ae47-e33a0983bc56
5602024-08-05T16:42:56.429ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_internal_dns_5145c3cc-86a8-4ca7-a5dd-d58654518f19
5612024-08-05T16:42:56.429ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5622024-08-05T16:42:56.429ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5632024-08-05T16:42:56.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 8734834 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55797 req_id = a6b6629a-566b-4f96-ac03-be2d8844e230 response_code = 204 uri = /omicron-zones
5642024-08-05T16:42:56.431ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5652024-08-05T16:42:56.484ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5662024-08-05T16:42:56.536ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5672024-08-05T16:42:56.587ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:591
5682024-08-05T16:42:56.587ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5692024-08-05T16:42:56.619ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5702024-08-05T16:42:56.620ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5712024-08-05T16:42:56.620ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5722024-08-05T16:42:56.649ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
5732024-08-05T16:42:56.649ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63984
5742024-08-05T16:42:56.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5752024-08-05T16:42:56.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5762024-08-05T16:42:56.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5772024-08-05T16:42:56.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
5782024-08-05T16:42:56.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
5792024-08-05T16:42:56.688ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5802024-08-05T16:42:56.705ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5812024-08-05T16:42:56.705ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
5822024-08-05T16:42:56.705ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
5832024-08-05T16:42:56.705ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5842024-08-05T16:42:56.716ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5852024-08-05T16:42:56.749ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5862024-08-05T16:42:56.749ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5872024-08-05T16:42:56.749ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5882024-08-05T16:42:56.778ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 247, 71]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5892024-08-05T16:42:56.778ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = boundary_ntp
5902024-08-05T16:42:56.779ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5912024-08-05T16:42:56.779ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5922024-08-05T16:42:56.779ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
5932024-08-05T16:42:56.780ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
5942024-08-05T16:42:56.804ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b file = illumos-utils/src/zone.rs:316
5952024-08-05T16:42:56.822ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b file = illumos-utils/src/zone.rs:354
5962024-08-05T16:42:57.441ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5972024-08-05T16:42:57.442ZINFOSledAgent (ServiceManager): Booting oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
5982024-08-05T16:42:57.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 67.623322ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
5992024-08-05T16:42:57.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 118.200897ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6002024-08-05T16:42:57.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 240.495868ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6012024-08-05T16:42:58.054ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 274.425397ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6022024-08-05T16:42:58.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 841.624352ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6032024-08-05T16:42:59.190ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 1.014178632s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6042024-08-05T16:43:00.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 1.105422431s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6052024-08-05T16:43:01.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 965.86147ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6062024-08-05T16:43:02.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b") failed: Property not found. retry in 741.660638ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6072024-08-05T16:43:03.068ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b
6082024-08-05T16:43:03.068ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6092024-08-05T16:43:03.069ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6102024-08-05T16:43:03.069ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 6420061 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:63984 req_id = 96112238-7f5c-46cf-b341-847bf29225df response_code = 204 uri = /omicron-zones
6112024-08-05T16:43:03.072ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:632
6122024-08-05T16:43:03.101ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6132024-08-05T16:43:03.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60121
6142024-08-05T16:43:03.111ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3650
6152024-08-05T16:43:03.111ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 9439 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60121 req_id = cdce478e-b658-4641-a44c-17103bcf5280 response_code = 500 uri = /timesync
6162024-08-05T16:43:03.114ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6172024-08-05T16:43:03.223ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6182024-08-05T16:43:03.223ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43024
6192024-08-05T16:43:07.232ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3650
6202024-08-05T16:43:07.232ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 4008815 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43024 req_id = 22887bbc-a860-4b20-ad34-a3321f55e0df response_code = 500 uri = /timesync
6212024-08-05T16:43:07.235ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6222024-08-05T16:43:07.347ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6232024-08-05T16:43:07.347ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65434
6242024-08-05T16:43:07.357ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3650
6252024-08-05T16:43:07.357ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 9409 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65434 req_id = 657298ce-b31a-4e61-9605-b9ddb8f12b3d response_code = 500 uri = /timesync
6262024-08-05T16:43:07.359ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6272024-08-05T16:43:07.512ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6282024-08-05T16:43:07.512ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47146
6292024-08-05T16:43:10.656ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_5b6cee62-004c-4843-9e16-c25d2b0c743b': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3650
6302024-08-05T16:43:10.657ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 4008670 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47146 req_id = 9a4d791f-780a-4f29-9b1a-8a7e8504fd08 response_code = 500 uri = /timesync
6312024-08-05T16:43:10.659ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6322024-08-05T16:43:10.832ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6332024-08-05T16:43:10.832ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39239
6342024-08-05T16:43:10.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 20159 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39239 req_id = 81d65d02-aa49-4309-b37e-720cf1b37677 response_code = 200 uri = /timesync
6352024-08-05T16:43:10.857ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1722876189.9402506, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6362024-08-05T16:43:10.857ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1230
6372024-08-05T16:43:10.887ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6382024-08-05T16:43:10.887ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55017
6392024-08-05T16:43:10.887ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6402024-08-05T16:43:10.887ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6412024-08-05T16:43:10.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6422024-08-05T16:43:10.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
6432024-08-05T16:43:10.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
6442024-08-05T16:43:10.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6452024-08-05T16:43:10.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6462024-08-05T16:43:10.946ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6472024-08-05T16:43:11.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6482024-08-05T16:43:11.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6492024-08-05T16:43:11.071ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6502024-08-05T16:43:11.071ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6512024-08-05T16:43:11.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6522024-08-05T16:43:11.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6532024-08-05T16:43:11.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
6542024-08-05T16:43:11.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6552024-08-05T16:43:11.262ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6562024-08-05T16:43:11.262ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
6572024-08-05T16:43:11.262ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
6582024-08-05T16:43:11.262ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6592024-08-05T16:43:11.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6602024-08-05T16:43:11.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6612024-08-05T16:43:11.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6622024-08-05T16:43:11.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6632024-08-05T16:43:11.346ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
6642024-08-05T16:43:11.436ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba file = illumos-utils/src/zone.rs:316
6652024-08-05T16:43:11.440ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53 file = illumos-utils/src/zone.rs:316
6662024-08-05T16:43:11.444ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87 file = illumos-utils/src/zone.rs:316
6672024-08-05T16:43:11.448ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57 file = illumos-utils/src/zone.rs:316
6682024-08-05T16:43:11.452ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a file = illumos-utils/src/zone.rs:316
6692024-08-05T16:43:11.457ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba file = illumos-utils/src/zone.rs:354
6702024-08-05T16:43:11.462ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53 file = illumos-utils/src/zone.rs:354
6712024-08-05T16:43:11.466ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87 file = illumos-utils/src/zone.rs:354
6722024-08-05T16:43:11.470ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57 file = illumos-utils/src/zone.rs:354
6732024-08-05T16:43:11.475ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a file = illumos-utils/src/zone.rs:354
6742024-08-05T16:43:13.614ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 374.536001ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
6752024-08-05T16:43:14.351ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='3c470f26-8422-4139-9f7f-81a0d84becba'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6762024-08-05T16:43:14.352ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
6772024-08-05T16:43:14.366ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='eed4ad2f-7765-43e1-8943-ea458ed22e57'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6782024-08-05T16:43:14.366ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
6792024-08-05T16:43:14.395ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='acae5a41-5da9-420f-9bc8-154e4abdaf87'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6802024-08-05T16:43:14.395ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
6812024-08-05T16:43:14.400ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='902ea001-1913-4130-9979-87a8d3db1f53'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6822024-08-05T16:43:14.400ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
6832024-08-05T16:43:14.494ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='b2f11b30-44fe-4b76-98c0-1891b1b0971a'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6842024-08-05T16:43:14.494ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
6852024-08-05T16:43:14.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 26.780164ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
6862024-08-05T16:43:14.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 30.398258ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
6872024-08-05T16:43:14.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 65.52641ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
6882024-08-05T16:43:14.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 65.553682ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
6892024-08-05T16:43:14.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 89.995037ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
6902024-08-05T16:43:14.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 57.71122ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
6912024-08-05T16:43:14.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 60.791258ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
6922024-08-05T16:43:15.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 112.498824ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
6932024-08-05T16:43:15.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 145.267808ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
6942024-08-05T16:43:15.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 203.023334ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
6952024-08-05T16:43:15.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 276.169191ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
6962024-08-05T16:43:15.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 228.009776ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
6972024-08-05T16:43:15.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 238.382755ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
6982024-08-05T16:43:16.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 401.283721ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
6992024-08-05T16:43:16.287ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 398.782537ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7002024-08-05T16:43:16.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 61.970725ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7012024-08-05T16:43:16.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 494.863254ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7022024-08-05T16:43:16.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 236.825194ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7032024-08-05T16:43:16.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 837.212067ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7042024-08-05T16:43:16.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 756.133841ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7052024-08-05T16:43:16.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 125.567175ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7062024-08-05T16:43:16.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 278.82169ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7072024-08-05T16:43:16.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 1.065798478s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7082024-08-05T16:43:17.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 1.267241012s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7092024-08-05T16:43:17.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 210.585035ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7102024-08-05T16:43:17.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 762.980308ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7112024-08-05T16:43:17.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 519.254827ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7122024-08-05T16:43:17.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 605.333726ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7132024-08-05T16:43:18.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 1.021059132s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7142024-08-05T16:43:18.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 608.389297ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7152024-08-05T16:43:18.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 1.118393339s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7162024-08-05T16:43:18.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 763.027699ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7172024-08-05T16:43:18.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 1.021733309s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7182024-08-05T16:43:18.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 894.641188ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7192024-08-05T16:43:19.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 888.055016ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7202024-08-05T16:43:19.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 578.393447ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7212024-08-05T16:43:19.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 928.379446ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7222024-08-05T16:43:19.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 1.220228438s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7232024-08-05T16:43:19.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 614.766779ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7242024-08-05T16:43:19.681ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 714.805206ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7252024-08-05T16:43:19.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 829.918608ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7262024-08-05T16:43:20.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 969.542375ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7272024-08-05T16:43:20.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 926.738786ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7282024-08-05T16:43:20.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 968.186512ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7292024-08-05T16:43:20.711ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 1.090332609s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7302024-08-05T16:43:20.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 649.388012ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7312024-08-05T16:43:21.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 941.901562ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7322024-08-05T16:43:21.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 926.465875ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7332024-08-05T16:43:21.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 547.68534ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7342024-08-05T16:43:21.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 608.308828ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7352024-08-05T16:43:21.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 1.159070816s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7362024-08-05T16:43:21.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 824.908414ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7372024-08-05T16:43:22.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 1.105285073s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7382024-08-05T16:43:22.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 501.622866ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7392024-08-05T16:43:22.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 1.432271208s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7402024-08-05T16:43:22.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 974.119002ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7412024-08-05T16:43:22.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 1.19824924s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7422024-08-05T16:43:22.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 766.116091ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7432024-08-05T16:43:23.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 869.5578ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7442024-08-05T16:43:23.193ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65191
7452024-08-05T16:43:23.194ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:65191 req_id = 938e0ca9-7f8a-43c6-ac54-a5fc176a7e84 response_code = 200 uri = /baseboard
7462024-08-05T16:43:23.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87") failed: Property not found. retry in 1.206309915s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7472024-08-05T16:43:23.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 975.655348ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7482024-08-05T16:43:23.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba") failed: Property not found. retry in 977.411288ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7492024-08-05T16:43:23.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53") failed: Property not found. retry in 1.462498023s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7502024-08-05T16:43:24.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a") failed: Property not found. retry in 626.07511ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7512024-08-05T16:43:24.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 1.077199798s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7522024-08-05T16:43:25.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 1.421728126s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7532024-08-05T16:43:27.171ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 1.352387763s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7542024-08-05T16:43:28.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 1.244689885s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7552024-08-05T16:43:29.065ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 461.754385ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
7562024-08-05T16:43:29.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 713.411936ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7572024-08-05T16:43:30.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 1.480677947s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7582024-08-05T16:43:32.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 1.087141234s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7592024-08-05T16:43:33.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 787.566197ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7602024-08-05T16:43:33.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57") failed: Property not found. retry in 1.181913108s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7612024-08-05T16:43:35.127ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_cockroachdb_3c470f26-8422-4139-9f7f-81a0d84becba
7622024-08-05T16:43:35.127ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_cockroachdb_902ea001-1913-4130-9979-87a8d3db1f53
7632024-08-05T16:43:35.127ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_cockroachdb_acae5a41-5da9-420f-9bc8-154e4abdaf87
7642024-08-05T16:43:35.127ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_cockroachdb_eed4ad2f-7765-43e1-8943-ea458ed22e57
7652024-08-05T16:43:35.127ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_cockroachdb_b2f11b30-44fe-4b76-98c0-1891b1b0971a
7662024-08-05T16:43:35.127ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7672024-08-05T16:43:35.128ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7682024-08-05T16:43:35.129ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 24182462 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55017 req_id = 351cf85d-c913-4a8c-b8b6-7d60b0d7120b response_code = 204 uri = /omicron-zones
7692024-08-05T16:43:35.164ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39257
7702024-08-05T16:43:35.203ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::3]:32221 file = sled-agent/src/services.rs:3484
7712024-08-05T16:43:35.461ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3502
7722024-08-05T16:43:49.996ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3525
7732024-08-05T16:43:49.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 14790851 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:39257 req_id = a698c01f-9ce1-4c43-836d-ecb834147450 response_code = 204 uri = /cockroachdb
7742024-08-05T16:43:50.000ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 507.291197ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
7752024-08-05T16:43:50.031ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7762024-08-05T16:43:50.031ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59148
7772024-08-05T16:43:50.031ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7782024-08-05T16:43:50.031ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7792024-08-05T16:43:50.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7802024-08-05T16:43:50.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
7812024-08-05T16:43:50.074ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7822024-08-05T16:43:50.075ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7832024-08-05T16:43:50.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7842024-08-05T16:43:50.097ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7852024-08-05T16:43:50.116ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7862024-08-05T16:43:50.116ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7872024-08-05T16:43:50.135ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7882024-08-05T16:43:50.135ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7892024-08-05T16:43:50.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7902024-08-05T16:43:50.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7912024-08-05T16:43:50.173ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
7922024-08-05T16:43:50.173ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
7932024-08-05T16:43:50.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7942024-08-05T16:43:50.194ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
7952024-08-05T16:43:50.343ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
7962024-08-05T16:43:50.343ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
7972024-08-05T16:43:50.474ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
7982024-08-05T16:43:50.474ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
7992024-08-05T16:43:50.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8002024-08-05T16:43:50.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
8012024-08-05T16:43:50.790ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8022024-08-05T16:43:50.790ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
8032024-08-05T16:43:50.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8042024-08-05T16:43:50.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
8052024-08-05T16:43:51.058ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8062024-08-05T16:43:51.058ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
8072024-08-05T16:43:51.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8082024-08-05T16:43:51.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
8092024-08-05T16:43:51.303ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8102024-08-05T16:43:51.303ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
8112024-08-05T16:43:51.436ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8122024-08-05T16:43:51.436ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
8132024-08-05T16:43:51.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8142024-08-05T16:43:51.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
8152024-08-05T16:43:51.738ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8162024-08-05T16:43:51.738ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
8172024-08-05T16:43:51.878ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8182024-08-05T16:43:51.878ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
8192024-08-05T16:43:51.878ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
8202024-08-05T16:43:51.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8212024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
8222024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
8232024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
8242024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
8252024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
8262024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
8272024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
8282024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
8292024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
8302024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
8312024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
8322024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
8332024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
8342024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
8352024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
8362024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
8372024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
8382024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
8392024-08-05T16:43:52.000ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
8402024-08-05T16:43:52.000ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 1963668 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 1153b55a-081c-4b24-ab56-db62e8a5a6b4 response_code = 500 uri = /omicron-zones
8412024-08-05T16:43:52.000ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "1153b55a-081c-4b24-ab56-db62e8a5a6b4", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "1153b55a-081c-4b24-ab56-db62e8a5a6b4" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 131.947908ms sled_address = [fd00:1122:3344:101::1]:12345
8422024-08-05T16:43:52.133ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8432024-08-05T16:43:52.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8442024-08-05T16:43:52.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8452024-08-05T16:43:52.155ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8462024-08-05T16:43:52.155ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
8472024-08-05T16:43:52.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8482024-08-05T16:43:52.177ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8492024-08-05T16:43:52.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8502024-08-05T16:43:52.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8512024-08-05T16:43:52.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8522024-08-05T16:43:52.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8532024-08-05T16:43:52.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8542024-08-05T16:43:52.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8552024-08-05T16:43:52.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8562024-08-05T16:43:52.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8572024-08-05T16:43:52.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
8582024-08-05T16:43:52.271ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8592024-08-05T16:43:52.290ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8602024-08-05T16:43:52.290ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8612024-08-05T16:43:52.309ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
8622024-08-05T16:43:52.309ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8632024-08-05T16:43:52.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
8642024-08-05T16:43:52.330ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8652024-08-05T16:43:52.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8662024-08-05T16:43:52.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
8672024-08-05T16:43:52.369ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8682024-08-05T16:43:52.369ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
8692024-08-05T16:43:52.389ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8702024-08-05T16:43:52.389ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
8712024-08-05T16:43:52.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8722024-08-05T16:43:52.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
8732024-08-05T16:43:52.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8742024-08-05T16:43:52.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
8752024-08-05T16:43:52.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8762024-08-05T16:43:52.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
8772024-08-05T16:43:52.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8782024-08-05T16:43:52.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
8792024-08-05T16:43:52.484ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8802024-08-05T16:43:52.484ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
8812024-08-05T16:43:52.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
8822024-08-05T16:43:52.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
8832024-08-05T16:43:52.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8842024-08-05T16:43:52.524ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
8852024-08-05T16:43:52.524ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
8862024-08-05T16:43:52.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
8872024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
8882024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
8892024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
8902024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
8912024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
8922024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
8932024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
8942024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
8952024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
8962024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
8972024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
8982024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
8992024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
9002024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
9012024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
9022024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
9032024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
9042024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
9052024-08-05T16:43:52.623ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
9062024-08-05T16:43:52.623ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 488719 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 964ac0cc-7b55-4fdd-b25e-82c932a006d0 response_code = 500 uri = /omicron-zones
9072024-08-05T16:43:52.623ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "964ac0cc-7b55-4fdd-b25e-82c932a006d0", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "964ac0cc-7b55-4fdd-b25e-82c932a006d0" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 151.60857ms sled_address = [fd00:1122:3344:101::1]:12345
9082024-08-05T16:43:52.776ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9092024-08-05T16:43:52.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9102024-08-05T16:43:52.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9112024-08-05T16:43:52.799ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9122024-08-05T16:43:52.799ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
9132024-08-05T16:43:52.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9142024-08-05T16:43:52.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9152024-08-05T16:43:52.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9162024-08-05T16:43:52.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9172024-08-05T16:43:52.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9182024-08-05T16:43:52.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9192024-08-05T16:43:52.879ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9202024-08-05T16:43:52.879ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9212024-08-05T16:43:52.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9222024-08-05T16:43:52.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9232024-08-05T16:43:52.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9242024-08-05T16:43:52.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9252024-08-05T16:43:52.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9262024-08-05T16:43:52.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9272024-08-05T16:43:52.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9282024-08-05T16:43:52.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9292024-08-05T16:43:52.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
9302024-08-05T16:43:52.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9312024-08-05T16:43:52.997ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9322024-08-05T16:43:52.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
9332024-08-05T16:43:53.016ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9342024-08-05T16:43:53.016ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
9352024-08-05T16:43:53.036ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9362024-08-05T16:43:53.036ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
9372024-08-05T16:43:53.055ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9382024-08-05T16:43:53.055ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
9392024-08-05T16:43:53.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9402024-08-05T16:43:53.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
9412024-08-05T16:43:53.091ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9422024-08-05T16:43:53.091ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
9432024-08-05T16:43:53.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9442024-08-05T16:43:53.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
9452024-08-05T16:43:53.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9462024-08-05T16:43:53.131ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
9472024-08-05T16:43:53.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
9482024-08-05T16:43:53.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
9492024-08-05T16:43:53.172ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9502024-08-05T16:43:53.172ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
9512024-08-05T16:43:53.172ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
9522024-08-05T16:43:53.172ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
9532024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
9542024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
9552024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
9562024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
9572024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
9582024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
9592024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
9602024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
9612024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
9622024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
9632024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
9642024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
9652024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
9662024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
9672024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
9682024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
9692024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
9702024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
9712024-08-05T16:43:53.269ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
9722024-08-05T16:43:53.269ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 491801 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 77424964-b71d-4a4b-b89e-cab15db65585 response_code = 500 uri = /omicron-zones
9732024-08-05T16:43:53.269ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "77424964-b71d-4a4b-b89e-cab15db65585", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "77424964-b71d-4a4b-b89e-cab15db65585" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 78.889799ms sled_address = [fd00:1122:3344:101::1]:12345
9742024-08-05T16:43:53.309ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47139
9752024-08-05T16:43:53.309ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47139 req_id = eba2179c-f6a8-469c-bf08-daea5a007c6c response_code = 200 uri = /baseboard
9762024-08-05T16:43:53.349ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9772024-08-05T16:43:53.350ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9782024-08-05T16:43:53.350ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9792024-08-05T16:43:53.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9802024-08-05T16:43:53.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
9812024-08-05T16:43:53.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9822024-08-05T16:43:53.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9832024-08-05T16:43:53.410ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9842024-08-05T16:43:53.410ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9852024-08-05T16:43:53.433ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9862024-08-05T16:43:53.433ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9872024-08-05T16:43:53.454ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9882024-08-05T16:43:53.454ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9892024-08-05T16:43:53.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9902024-08-05T16:43:53.475ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9912024-08-05T16:43:53.496ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
9922024-08-05T16:43:53.496ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9932024-08-05T16:43:53.516ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9942024-08-05T16:43:53.516ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9952024-08-05T16:43:53.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
9962024-08-05T16:43:53.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9972024-08-05T16:43:53.556ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
9982024-08-05T16:43:53.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9992024-08-05T16:43:53.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10002024-08-05T16:43:53.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
10012024-08-05T16:43:53.594ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10022024-08-05T16:43:53.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
10032024-08-05T16:43:53.615ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10042024-08-05T16:43:53.615ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
10052024-08-05T16:43:53.635ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10062024-08-05T16:43:53.635ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
10072024-08-05T16:43:53.655ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10082024-08-05T16:43:53.655ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
10092024-08-05T16:43:53.673ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10102024-08-05T16:43:53.673ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
10112024-08-05T16:43:53.691ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10122024-08-05T16:43:53.691ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
10132024-08-05T16:43:53.709ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10142024-08-05T16:43:53.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
10152024-08-05T16:43:53.727ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10162024-08-05T16:43:53.727ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
10172024-08-05T16:43:53.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10182024-08-05T16:43:53.749ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
10192024-08-05T16:43:53.749ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
10202024-08-05T16:43:53.750ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10212024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
10222024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
10232024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
10242024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
10252024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
10262024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
10272024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
10282024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
10292024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
10302024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
10312024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
10322024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
10332024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
10342024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
10352024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
10362024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
10372024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
10382024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
10392024-08-05T16:43:53.846ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
10402024-08-05T16:43:53.846ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 495556 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 6ee9034d-ec5b-4f10-a903-a759d9ee1b3e response_code = 500 uri = /omicron-zones
10412024-08-05T16:43:53.846ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6ee9034d-ec5b-4f10-a903-a759d9ee1b3e", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6ee9034d-ec5b-4f10-a903-a759d9ee1b3e" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 230.952503ms sled_address = [fd00:1122:3344:101::1]:12345
10422024-08-05T16:43:54.080ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10432024-08-05T16:43:54.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10442024-08-05T16:43:54.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10452024-08-05T16:43:54.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10462024-08-05T16:43:54.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
10472024-08-05T16:43:54.122ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10482024-08-05T16:43:54.122ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10492024-08-05T16:43:54.143ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10502024-08-05T16:43:54.143ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10512024-08-05T16:43:54.162ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10522024-08-05T16:43:54.162ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10532024-08-05T16:43:54.183ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10542024-08-05T16:43:54.183ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10552024-08-05T16:43:54.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10562024-08-05T16:43:54.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10572024-08-05T16:43:54.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
10582024-08-05T16:43:54.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10592024-08-05T16:43:54.240ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10602024-08-05T16:43:54.240ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10612024-08-05T16:43:54.259ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
10622024-08-05T16:43:54.259ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10632024-08-05T16:43:54.280ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
10642024-08-05T16:43:54.280ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10652024-08-05T16:43:54.300ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10662024-08-05T16:43:54.300ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
10672024-08-05T16:43:54.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10682024-08-05T16:43:54.319ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
10692024-08-05T16:43:54.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10702024-08-05T16:43:54.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
10712024-08-05T16:43:54.358ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10722024-08-05T16:43:54.358ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
10732024-08-05T16:43:54.378ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10742024-08-05T16:43:54.378ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
10752024-08-05T16:43:54.398ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10762024-08-05T16:43:54.398ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
10772024-08-05T16:43:54.417ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10782024-08-05T16:43:54.417ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
10792024-08-05T16:43:54.437ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10802024-08-05T16:43:54.437ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
10812024-08-05T16:43:54.457ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
10822024-08-05T16:43:54.457ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
10832024-08-05T16:43:54.476ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10842024-08-05T16:43:54.476ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
10852024-08-05T16:43:54.476ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
10862024-08-05T16:43:54.476ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
10872024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
10882024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
10892024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
10902024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
10912024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
10922024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
10932024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
10942024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
10952024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
10962024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
10972024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
10982024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
10992024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
11002024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
11012024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
11022024-08-05T16:43:54.571ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
11032024-08-05T16:43:54.572ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
11042024-08-05T16:43:54.572ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
11052024-08-05T16:43:54.572ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
11062024-08-05T16:43:54.572ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 490637 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 34d9b261-bc03-4d9c-832e-73d7ec98651c response_code = 500 uri = /omicron-zones
11072024-08-05T16:43:54.572ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "34d9b261-bc03-4d9c-832e-73d7ec98651c", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "34d9b261-bc03-4d9c-832e-73d7ec98651c" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 297.63195ms sled_address = [fd00:1122:3344:101::1]:12345
11082024-08-05T16:43:54.872ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11092024-08-05T16:43:54.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11102024-08-05T16:43:54.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11112024-08-05T16:43:54.896ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11122024-08-05T16:43:54.896ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
11132024-08-05T16:43:54.914ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11142024-08-05T16:43:54.914ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11152024-08-05T16:43:54.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11162024-08-05T16:43:54.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11172024-08-05T16:43:54.955ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11182024-08-05T16:43:54.955ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11192024-08-05T16:43:54.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11202024-08-05T16:43:54.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11212024-08-05T16:43:54.992ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11222024-08-05T16:43:54.992ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11232024-08-05T16:43:55.011ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11242024-08-05T16:43:55.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11252024-08-05T16:43:55.032ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11262024-08-05T16:43:55.032ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11272024-08-05T16:43:55.051ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11282024-08-05T16:43:55.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11292024-08-05T16:43:55.072ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
11302024-08-05T16:43:55.072ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11312024-08-05T16:43:55.092ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11322024-08-05T16:43:55.092ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
11332024-08-05T16:43:55.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11342024-08-05T16:43:55.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
11352024-08-05T16:43:55.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11362024-08-05T16:43:55.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
11372024-08-05T16:43:55.152ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11382024-08-05T16:43:55.152ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
11392024-08-05T16:43:55.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11402024-08-05T16:43:55.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
11412024-08-05T16:43:55.191ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11422024-08-05T16:43:55.191ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
11432024-08-05T16:43:55.209ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11442024-08-05T16:43:55.209ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
11452024-08-05T16:43:55.228ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11462024-08-05T16:43:55.228ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
11472024-08-05T16:43:55.250ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11482024-08-05T16:43:55.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
11492024-08-05T16:43:55.269ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11502024-08-05T16:43:55.269ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
11512024-08-05T16:43:55.269ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
11522024-08-05T16:43:55.269ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
11532024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
11542024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
11552024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
11562024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
11572024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
11582024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
11592024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
11602024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
11612024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
11622024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
11632024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
11642024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
11652024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
11662024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
11672024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
11682024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
11692024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
11702024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
11712024-08-05T16:43:55.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
11722024-08-05T16:43:55.364ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 490986 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = a698c032-9f09-4711-9ed5-4b5c7aa799bb response_code = 500 uri = /omicron-zones
11732024-08-05T16:43:55.365ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a698c032-9f09-4711-9ed5-4b5c7aa799bb", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a698c032-9f09-4711-9ed5-4b5c7aa799bb" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 229.168562ms sled_address = [fd00:1122:3344:101::1]:12345
11742024-08-05T16:43:55.596ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11752024-08-05T16:43:55.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11762024-08-05T16:43:55.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11772024-08-05T16:43:55.619ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11782024-08-05T16:43:55.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
11792024-08-05T16:43:55.638ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11802024-08-05T16:43:55.638ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11812024-08-05T16:43:55.659ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11822024-08-05T16:43:55.659ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11832024-08-05T16:43:55.678ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11842024-08-05T16:43:55.678ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11852024-08-05T16:43:55.697ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11862024-08-05T16:43:55.697ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11872024-08-05T16:43:55.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11882024-08-05T16:43:55.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11892024-08-05T16:43:55.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
11902024-08-05T16:43:55.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11912024-08-05T16:43:55.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11922024-08-05T16:43:55.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11932024-08-05T16:43:55.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
11942024-08-05T16:43:55.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11952024-08-05T16:43:55.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
11962024-08-05T16:43:55.797ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11972024-08-05T16:43:55.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
11982024-08-05T16:43:55.816ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
11992024-08-05T16:43:55.835ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12002024-08-05T16:43:55.835ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
12012024-08-05T16:43:55.853ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12022024-08-05T16:43:55.853ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
12032024-08-05T16:43:55.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12042024-08-05T16:43:55.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
12052024-08-05T16:43:55.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12062024-08-05T16:43:55.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
12072024-08-05T16:43:55.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12082024-08-05T16:43:55.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
12092024-08-05T16:43:55.936ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12102024-08-05T16:43:55.936ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
12112024-08-05T16:43:55.955ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12122024-08-05T16:43:55.955ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
12132024-08-05T16:43:55.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12142024-08-05T16:43:55.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
12152024-08-05T16:43:55.993ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12162024-08-05T16:43:55.993ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
12172024-08-05T16:43:55.993ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
12182024-08-05T16:43:55.993ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12192024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
12202024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
12212024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
12222024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
12232024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
12242024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
12252024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
12262024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
12272024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
12282024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
12292024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
12302024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
12312024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
12322024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
12332024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
12342024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
12352024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
12362024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
12372024-08-05T16:43:56.091ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
12382024-08-05T16:43:56.092ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 494402 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = eeabf52b-d03b-4d4b-9511-52317510f7d8 response_code = 500 uri = /omicron-zones
12392024-08-05T16:43:56.092ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "eeabf52b-d03b-4d4b-9511-52317510f7d8", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "eeabf52b-d03b-4d4b-9511-52317510f7d8" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 417.393124ms sled_address = [fd00:1122:3344:101::1]:12345
12402024-08-05T16:43:56.512ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12412024-08-05T16:43:56.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12422024-08-05T16:43:56.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12432024-08-05T16:43:56.534ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12442024-08-05T16:43:56.534ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
12452024-08-05T16:43:56.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12462024-08-05T16:43:56.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12472024-08-05T16:43:56.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12482024-08-05T16:43:56.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12492024-08-05T16:43:56.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12502024-08-05T16:43:56.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12512024-08-05T16:43:56.620ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12522024-08-05T16:43:56.620ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12532024-08-05T16:43:56.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12542024-08-05T16:43:56.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12552024-08-05T16:43:56.659ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
12562024-08-05T16:43:56.659ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12572024-08-05T16:43:56.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12582024-08-05T16:43:56.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12592024-08-05T16:43:56.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
12602024-08-05T16:43:56.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12612024-08-05T16:43:56.719ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
12622024-08-05T16:43:56.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12632024-08-05T16:43:56.739ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12642024-08-05T16:43:56.739ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
12652024-08-05T16:43:56.760ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12662024-08-05T16:43:56.760ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
12672024-08-05T16:43:56.779ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12682024-08-05T16:43:56.779ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
12692024-08-05T16:43:56.800ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12702024-08-05T16:43:56.800ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
12712024-08-05T16:43:56.821ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12722024-08-05T16:43:56.821ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
12732024-08-05T16:43:56.841ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12742024-08-05T16:43:56.842ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
12752024-08-05T16:43:56.862ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12762024-08-05T16:43:56.862ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
12772024-08-05T16:43:56.881ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12782024-08-05T16:43:56.881ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
12792024-08-05T16:43:56.902ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
12802024-08-05T16:43:56.902ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
12812024-08-05T16:43:56.922ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12822024-08-05T16:43:56.922ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
12832024-08-05T16:43:56.922ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
12842024-08-05T16:43:56.922ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
12852024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
12862024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
12872024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
12882024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
12892024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
12902024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
12912024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
12922024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
12932024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
12942024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
12952024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
12962024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
12972024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
12982024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
12992024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
13002024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
13012024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
13022024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
13032024-08-05T16:43:57.025ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
13042024-08-05T16:43:57.025ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 511215 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = dc71e953-3718-4311-b0c4-d25a057af8e2 response_code = 500 uri = /omicron-zones
13052024-08-05T16:43:57.025ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dc71e953-3718-4311-b0c4-d25a057af8e2", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:57 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dc71e953-3718-4311-b0c4-d25a057af8e2" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 299.606921ms sled_address = [fd00:1122:3344:101::1]:12345
13062024-08-05T16:43:57.328ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13072024-08-05T16:43:57.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13082024-08-05T16:43:57.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13092024-08-05T16:43:57.352ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13102024-08-05T16:43:57.352ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
13112024-08-05T16:43:57.372ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13122024-08-05T16:43:57.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13132024-08-05T16:43:57.390ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13142024-08-05T16:43:57.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13152024-08-05T16:43:57.411ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13162024-08-05T16:43:57.411ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13172024-08-05T16:43:57.432ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13182024-08-05T16:43:57.432ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13192024-08-05T16:43:57.454ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13202024-08-05T16:43:57.454ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13212024-08-05T16:43:57.474ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13222024-08-05T16:43:57.474ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13232024-08-05T16:43:57.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13242024-08-05T16:43:57.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13252024-08-05T16:43:57.512ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13262024-08-05T16:43:57.512ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13272024-08-05T16:43:57.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
13282024-08-05T16:43:57.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13292024-08-05T16:43:57.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13302024-08-05T16:43:57.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
13312024-08-05T16:43:57.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13322024-08-05T16:43:57.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
13332024-08-05T16:43:57.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13342024-08-05T16:43:57.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
13352024-08-05T16:43:57.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13362024-08-05T16:43:57.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
13372024-08-05T16:43:57.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13382024-08-05T16:43:57.628ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
13392024-08-05T16:43:57.646ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13402024-08-05T16:43:57.646ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
13412024-08-05T16:43:57.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13422024-08-05T16:43:57.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
13432024-08-05T16:43:57.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13442024-08-05T16:43:57.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
13452024-08-05T16:43:57.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13462024-08-05T16:43:57.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
13472024-08-05T16:43:57.734ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13482024-08-05T16:43:57.734ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
13492024-08-05T16:43:57.734ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
13502024-08-05T16:43:57.734ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
13512024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
13522024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
13532024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
13542024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
13552024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
13562024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
13572024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
13582024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
13592024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
13602024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
13612024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
13622024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
13632024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
13642024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
13652024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
13662024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
13672024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
13682024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
13692024-08-05T16:43:57.830ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
13702024-08-05T16:43:57.830ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 501318 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 463a5426-0c6b-4a6f-b23d-57b8f1452ea2 response_code = 500 uri = /omicron-zones
13712024-08-05T16:43:57.830ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "463a5426-0c6b-4a6f-b23d-57b8f1452ea2", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:57 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "463a5426-0c6b-4a6f-b23d-57b8f1452ea2" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 475.581437ms sled_address = [fd00:1122:3344:101::1]:12345
13722024-08-05T16:43:58.309ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13732024-08-05T16:43:58.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13742024-08-05T16:43:58.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13752024-08-05T16:43:58.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13762024-08-05T16:43:58.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
13772024-08-05T16:43:58.352ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13782024-08-05T16:43:58.352ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13792024-08-05T16:43:58.373ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13802024-08-05T16:43:58.373ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13812024-08-05T16:43:58.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13822024-08-05T16:43:58.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13832024-08-05T16:43:58.410ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13842024-08-05T16:43:58.410ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13852024-08-05T16:43:58.430ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13862024-08-05T16:43:58.430ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13872024-08-05T16:43:58.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
13882024-08-05T16:43:58.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13892024-08-05T16:43:58.467ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13902024-08-05T16:43:58.467ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13912024-08-05T16:43:58.487ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
13922024-08-05T16:43:58.487ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13932024-08-05T16:43:58.506ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
13942024-08-05T16:43:58.506ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13952024-08-05T16:43:58.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13962024-08-05T16:43:58.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
13972024-08-05T16:43:58.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
13982024-08-05T16:43:58.545ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
13992024-08-05T16:43:58.563ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14002024-08-05T16:43:58.563ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
14012024-08-05T16:43:58.582ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14022024-08-05T16:43:58.582ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
14032024-08-05T16:43:58.602ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14042024-08-05T16:43:58.602ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
14052024-08-05T16:43:58.622ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14062024-08-05T16:43:58.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
14072024-08-05T16:43:58.643ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14082024-08-05T16:43:58.643ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
14092024-08-05T16:43:58.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14102024-08-05T16:43:58.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
14112024-08-05T16:43:58.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14122024-08-05T16:43:58.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
14132024-08-05T16:43:58.704ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14142024-08-05T16:43:58.704ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
14152024-08-05T16:43:58.704ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
14162024-08-05T16:43:58.704ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14172024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
14182024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
14192024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
14202024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
14212024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
14222024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
14232024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
14242024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
14252024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
14262024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
14272024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
14282024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
14292024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
14302024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
14312024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
14322024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
14332024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
14342024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
14352024-08-05T16:43:58.800ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
14362024-08-05T16:43:58.800ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 489458 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 05ed379d-b82e-4b40-9078-20de00baf85a response_code = 500 uri = /omicron-zones
14372024-08-05T16:43:58.800ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "05ed379d-b82e-4b40-9078-20de00baf85a", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "05ed379d-b82e-4b40-9078-20de00baf85a" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 458.479111ms sled_address = [fd00:1122:3344:101::1]:12345
14382024-08-05T16:43:59.262ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14392024-08-05T16:43:59.262ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14402024-08-05T16:43:59.262ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14412024-08-05T16:43:59.285ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14422024-08-05T16:43:59.285ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
14432024-08-05T16:43:59.305ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14442024-08-05T16:43:59.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14452024-08-05T16:43:59.324ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14462024-08-05T16:43:59.324ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14472024-08-05T16:43:59.343ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14482024-08-05T16:43:59.343ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14492024-08-05T16:43:59.363ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14502024-08-05T16:43:59.363ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14512024-08-05T16:43:59.382ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14522024-08-05T16:43:59.382ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14532024-08-05T16:43:59.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
14542024-08-05T16:43:59.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14552024-08-05T16:43:59.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14562024-08-05T16:43:59.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14572024-08-05T16:43:59.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
14582024-08-05T16:43:59.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14592024-08-05T16:43:59.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
14602024-08-05T16:43:59.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14612024-08-05T16:43:59.479ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14622024-08-05T16:43:59.479ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
14632024-08-05T16:43:59.499ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14642024-08-05T16:43:59.499ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
14652024-08-05T16:43:59.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14662024-08-05T16:43:59.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
14672024-08-05T16:43:59.537ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14682024-08-05T16:43:59.537ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
14692024-08-05T16:43:59.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14702024-08-05T16:43:59.555ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
14712024-08-05T16:43:59.575ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14722024-08-05T16:43:59.575ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
14732024-08-05T16:43:59.593ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14742024-08-05T16:43:59.593ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
14752024-08-05T16:43:59.612ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14762024-08-05T16:43:59.612ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
14772024-08-05T16:43:59.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
14782024-08-05T16:43:59.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
14792024-08-05T16:43:59.651ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14802024-08-05T16:43:59.651ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
14812024-08-05T16:43:59.651ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
14822024-08-05T16:43:59.651ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
14832024-08-05T16:43:59.746ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
14842024-08-05T16:43:59.746ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
14852024-08-05T16:43:59.746ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
14862024-08-05T16:43:59.746ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
14872024-08-05T16:43:59.746ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
14882024-08-05T16:43:59.746ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
14892024-08-05T16:43:59.746ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
14902024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
14912024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
14922024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
14932024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
14942024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
14952024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
14962024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
14972024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
14982024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
14992024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
15002024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
15012024-08-05T16:43:59.747ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
15022024-08-05T16:43:59.747ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 483270 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = a32214d9-5b62-4c32-9d26-11743c0cf571 response_code = 500 uri = /omicron-zones
15032024-08-05T16:43:59.747ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a32214d9-5b62-4c32-9d26-11743c0cf571", "content-length": "124", "date": "Mon, 05 Aug 2024 16:43:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a32214d9-5b62-4c32-9d26-11743c0cf571" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 584.521444ms sled_address = [fd00:1122:3344:101::1]:12345
15042024-08-05T16:44:00.335ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
15052024-08-05T16:44:00.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15062024-08-05T16:44:00.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15072024-08-05T16:44:00.358ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15082024-08-05T16:44:00.358ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
15092024-08-05T16:44:00.378ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15102024-08-05T16:44:00.378ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15112024-08-05T16:44:00.397ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15122024-08-05T16:44:00.397ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15132024-08-05T16:44:00.418ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15142024-08-05T16:44:00.418ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15152024-08-05T16:44:00.437ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15162024-08-05T16:44:00.437ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15172024-08-05T16:44:00.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15182024-08-05T16:44:00.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15192024-08-05T16:44:00.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15202024-08-05T16:44:00.475ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15212024-08-05T16:44:00.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15222024-08-05T16:44:00.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15232024-08-05T16:44:00.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15242024-08-05T16:44:00.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15252024-08-05T16:44:00.531ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
15262024-08-05T16:44:00.531ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15272024-08-05T16:44:00.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15282024-08-05T16:44:00.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
15292024-08-05T16:44:00.570ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15302024-08-05T16:44:00.570ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
15312024-08-05T16:44:00.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15322024-08-05T16:44:00.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
15332024-08-05T16:44:00.610ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15342024-08-05T16:44:00.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
15352024-08-05T16:44:00.630ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15362024-08-05T16:44:00.630ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
15372024-08-05T16:44:00.651ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15382024-08-05T16:44:00.651ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
15392024-08-05T16:44:00.671ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15402024-08-05T16:44:00.671ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
15412024-08-05T16:44:00.690ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15422024-08-05T16:44:00.690ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
15432024-08-05T16:44:00.709ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15442024-08-05T16:44:00.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
15452024-08-05T16:44:00.729ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15462024-08-05T16:44:00.729ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
15472024-08-05T16:44:00.729ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
15482024-08-05T16:44:00.729ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
15492024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
15502024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
15512024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
15522024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
15532024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
15542024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
15552024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
15562024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
15572024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
15582024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
15592024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
15602024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
15612024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
15622024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
15632024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
15642024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
15652024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
15662024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
15672024-08-05T16:44:00.829ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
15682024-08-05T16:44:00.829ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 492811 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = b43927c7-7631-40f0-9ea6-b15ba4077f44 response_code = 500 uri = /omicron-zones
15692024-08-05T16:44:00.829ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b43927c7-7631-40f0-9ea6-b15ba4077f44", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b43927c7-7631-40f0-9ea6-b15ba4077f44" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 753.966264ms sled_address = [fd00:1122:3344:101::1]:12345
15702024-08-05T16:44:01.587ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
15712024-08-05T16:44:01.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15722024-08-05T16:44:01.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15732024-08-05T16:44:01.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15742024-08-05T16:44:01.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
15752024-08-05T16:44:01.626ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15762024-08-05T16:44:01.626ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15772024-08-05T16:44:01.644ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15782024-08-05T16:44:01.644ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15792024-08-05T16:44:01.662ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15802024-08-05T16:44:01.662ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15812024-08-05T16:44:01.683ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15822024-08-05T16:44:01.683ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15832024-08-05T16:44:01.701ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15842024-08-05T16:44:01.701ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15852024-08-05T16:44:01.720ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
15862024-08-05T16:44:01.720ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15872024-08-05T16:44:01.740ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15882024-08-05T16:44:01.740ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15892024-08-05T16:44:01.759ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
15902024-08-05T16:44:01.759ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15912024-08-05T16:44:01.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
15922024-08-05T16:44:01.778ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15932024-08-05T16:44:01.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15942024-08-05T16:44:01.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
15952024-08-05T16:44:01.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15962024-08-05T16:44:01.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
15972024-08-05T16:44:01.831ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
15982024-08-05T16:44:01.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
15992024-08-05T16:44:01.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16002024-08-05T16:44:01.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
16012024-08-05T16:44:01.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16022024-08-05T16:44:01.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
16032024-08-05T16:44:01.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16042024-08-05T16:44:01.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
16052024-08-05T16:44:01.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16062024-08-05T16:44:01.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
16072024-08-05T16:44:01.924ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16082024-08-05T16:44:01.924ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
16092024-08-05T16:44:01.942ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16102024-08-05T16:44:01.942ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
16112024-08-05T16:44:01.961ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16122024-08-05T16:44:01.961ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
16132024-08-05T16:44:01.961ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
16142024-08-05T16:44:01.961ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16152024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
16162024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
16172024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
16182024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
16192024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
16202024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
16212024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
16222024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
16232024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
16242024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
16252024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
16262024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
16272024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
16282024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
16292024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
16302024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
16312024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
16322024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
16332024-08-05T16:44:02.056ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
16342024-08-05T16:44:02.057ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 468179 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = d143f344-9a5c-471e-9277-71af0db223c5 response_code = 500 uri = /omicron-zones
16352024-08-05T16:44:02.057ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d143f344-9a5c-471e-9277-71af0db223c5", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d143f344-9a5c-471e-9277-71af0db223c5" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 708.394841ms sled_address = [fd00:1122:3344:101::1]:12345
16362024-08-05T16:44:02.768ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
16372024-08-05T16:44:02.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16382024-08-05T16:44:02.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16392024-08-05T16:44:02.790ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16402024-08-05T16:44:02.790ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
16412024-08-05T16:44:02.811ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16422024-08-05T16:44:02.811ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16432024-08-05T16:44:02.829ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16442024-08-05T16:44:02.829ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16452024-08-05T16:44:02.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16462024-08-05T16:44:02.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16472024-08-05T16:44:02.865ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16482024-08-05T16:44:02.865ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16492024-08-05T16:44:02.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16502024-08-05T16:44:02.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16512024-08-05T16:44:02.903ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
16522024-08-05T16:44:02.903ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16532024-08-05T16:44:02.921ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16542024-08-05T16:44:02.921ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16552024-08-05T16:44:02.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
16562024-08-05T16:44:02.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16572024-08-05T16:44:02.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
16582024-08-05T16:44:02.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16592024-08-05T16:44:02.981ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16602024-08-05T16:44:02.981ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
16612024-08-05T16:44:02.999ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16622024-08-05T16:44:02.999ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
16632024-08-05T16:44:03.018ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16642024-08-05T16:44:03.018ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
16652024-08-05T16:44:03.037ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16662024-08-05T16:44:03.037ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
16672024-08-05T16:44:03.057ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16682024-08-05T16:44:03.057ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
16692024-08-05T16:44:03.078ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16702024-08-05T16:44:03.078ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
16712024-08-05T16:44:03.099ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16722024-08-05T16:44:03.099ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
16732024-08-05T16:44:03.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16742024-08-05T16:44:03.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
16752024-08-05T16:44:03.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
16762024-08-05T16:44:03.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
16772024-08-05T16:44:03.158ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16782024-08-05T16:44:03.158ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
16792024-08-05T16:44:03.158ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
16802024-08-05T16:44:03.158ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
16812024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
16822024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
16832024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
16842024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
16852024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
16862024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
16872024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
16882024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
16892024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
16902024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
16912024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
16922024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
16932024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
16942024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
16952024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
16962024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
16972024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
16982024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
16992024-08-05T16:44:03.252ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
17002024-08-05T16:44:03.252ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 482788 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 3e33f5b7-9f8c-47f1-b2c8-06e0dfdcd6fc response_code = 500 uri = /omicron-zones
17012024-08-05T16:44:03.253ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3e33f5b7-9f8c-47f1-b2c8-06e0dfdcd6fc", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3e33f5b7-9f8c-47f1-b2c8-06e0dfdcd6fc" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.008121169s sled_address = [fd00:1122:3344:101::1]:12345
17022024-08-05T16:44:04.265ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
17032024-08-05T16:44:04.266ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17042024-08-05T16:44:04.266ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17052024-08-05T16:44:04.289ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17062024-08-05T16:44:04.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
17072024-08-05T16:44:04.307ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17082024-08-05T16:44:04.307ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17092024-08-05T16:44:04.327ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17102024-08-05T16:44:04.327ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17112024-08-05T16:44:04.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17122024-08-05T16:44:04.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17132024-08-05T16:44:04.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17142024-08-05T16:44:04.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17152024-08-05T16:44:04.386ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17162024-08-05T16:44:04.386ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17172024-08-05T16:44:04.405ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17182024-08-05T16:44:04.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17192024-08-05T16:44:04.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17202024-08-05T16:44:04.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17212024-08-05T16:44:04.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17222024-08-05T16:44:04.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17232024-08-05T16:44:04.463ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
17242024-08-05T16:44:04.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17252024-08-05T16:44:04.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17262024-08-05T16:44:04.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
17272024-08-05T16:44:04.502ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17282024-08-05T16:44:04.502ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
17292024-08-05T16:44:04.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17302024-08-05T16:44:04.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
17312024-08-05T16:44:04.539ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17322024-08-05T16:44:04.539ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
17332024-08-05T16:44:04.560ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17342024-08-05T16:44:04.560ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
17352024-08-05T16:44:04.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17362024-08-05T16:44:04.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
17372024-08-05T16:44:04.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17382024-08-05T16:44:04.597ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
17392024-08-05T16:44:04.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17402024-08-05T16:44:04.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
17412024-08-05T16:44:04.634ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17422024-08-05T16:44:04.634ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
17432024-08-05T16:44:04.653ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17442024-08-05T16:44:04.653ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
17452024-08-05T16:44:04.653ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
17462024-08-05T16:44:04.653ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
17472024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
17482024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
17492024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
17502024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
17512024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
17522024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
17532024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
17542024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
17552024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
17562024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
17572024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
17582024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
17592024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
17602024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
17612024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
17622024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
17632024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
17642024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
17652024-08-05T16:44:04.752ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
17662024-08-05T16:44:04.752ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 485562 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 2c5a56b7-c7e9-4e2e-8b17-47af1789af0a response_code = 500 uri = /omicron-zones
17672024-08-05T16:44:04.752ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2c5a56b7-c7e9-4e2e-8b17-47af1789af0a", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:04 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2c5a56b7-c7e9-4e2e-8b17-47af1789af0a" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.701845153s sled_address = [fd00:1122:3344:101::1]:12345
17682024-08-05T16:44:05.588ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.167683206s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
17692024-08-05T16:44:06.460ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
17702024-08-05T16:44:06.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17712024-08-05T16:44:06.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17722024-08-05T16:44:06.484ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17732024-08-05T16:44:06.484ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
17742024-08-05T16:44:06.505ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17752024-08-05T16:44:06.505ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17762024-08-05T16:44:06.525ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17772024-08-05T16:44:06.525ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17782024-08-05T16:44:06.548ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17792024-08-05T16:44:06.548ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17802024-08-05T16:44:06.568ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17812024-08-05T16:44:06.568ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17822024-08-05T16:44:06.590ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17832024-08-05T16:44:06.590ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17842024-08-05T16:44:06.611ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
17852024-08-05T16:44:06.611ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17862024-08-05T16:44:06.631ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17872024-08-05T16:44:06.631ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17882024-08-05T16:44:06.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
17892024-08-05T16:44:06.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17902024-08-05T16:44:06.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
17912024-08-05T16:44:06.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17922024-08-05T16:44:06.697ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17932024-08-05T16:44:06.697ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
17942024-08-05T16:44:06.719ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17952024-08-05T16:44:06.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
17962024-08-05T16:44:06.742ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17972024-08-05T16:44:06.742ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
17982024-08-05T16:44:06.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
17992024-08-05T16:44:06.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
18002024-08-05T16:44:06.785ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18012024-08-05T16:44:06.785ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
18022024-08-05T16:44:06.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18032024-08-05T16:44:06.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
18042024-08-05T16:44:06.831ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18052024-08-05T16:44:06.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
18062024-08-05T16:44:06.852ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18072024-08-05T16:44:06.853ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
18082024-08-05T16:44:06.875ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18092024-08-05T16:44:06.875ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
18102024-08-05T16:44:06.895ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18112024-08-05T16:44:06.895ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
18122024-08-05T16:44:06.895ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
18132024-08-05T16:44:06.895ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18142024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
18152024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
18162024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
18172024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
18182024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
18192024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
18202024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
18212024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
18222024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
18232024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
18242024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
18252024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
18262024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
18272024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
18282024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
18292024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
18302024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
18312024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
18322024-08-05T16:44:06.994ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
18332024-08-05T16:44:06.995ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 532880 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 7880e5ec-643f-4849-8dae-1fe23ef4989d response_code = 500 uri = /omicron-zones
18342024-08-05T16:44:06.995ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7880e5ec-643f-4849-8dae-1fe23ef4989d", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:06 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7880e5ec-643f-4849-8dae-1fe23ef4989d" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.592005046s sled_address = [fd00:1122:3344:101::1]:12345
18352024-08-05T16:44:08.592ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
18362024-08-05T16:44:08.593ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
18372024-08-05T16:44:08.593ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18382024-08-05T16:44:08.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
18392024-08-05T16:44:08.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
18402024-08-05T16:44:08.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
18412024-08-05T16:44:08.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18422024-08-05T16:44:08.657ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
18432024-08-05T16:44:08.657ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18442024-08-05T16:44:08.675ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
18452024-08-05T16:44:08.675ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18462024-08-05T16:44:08.691ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
18472024-08-05T16:44:08.691ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18482024-08-05T16:44:08.711ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
18492024-08-05T16:44:08.711ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18502024-08-05T16:44:08.730ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
18512024-08-05T16:44:08.730ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18522024-08-05T16:44:08.752ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
18532024-08-05T16:44:08.752ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18542024-08-05T16:44:08.772ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
18552024-08-05T16:44:08.772ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18562024-08-05T16:44:08.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
18572024-08-05T16:44:08.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18582024-08-05T16:44:08.815ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18592024-08-05T16:44:08.815ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
18602024-08-05T16:44:08.836ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18612024-08-05T16:44:08.836ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
18622024-08-05T16:44:08.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18632024-08-05T16:44:08.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
18642024-08-05T16:44:08.877ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18652024-08-05T16:44:08.877ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
18662024-08-05T16:44:08.897ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18672024-08-05T16:44:08.897ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
18682024-08-05T16:44:08.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18692024-08-05T16:44:08.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
18702024-08-05T16:44:08.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18712024-08-05T16:44:08.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
18722024-08-05T16:44:08.960ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18732024-08-05T16:44:08.960ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
18742024-08-05T16:44:08.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
18752024-08-05T16:44:08.982ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
18762024-08-05T16:44:09.002ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18772024-08-05T16:44:09.002ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
18782024-08-05T16:44:09.003ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
18792024-08-05T16:44:09.003ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
18802024-08-05T16:44:09.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
18812024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
18822024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
18832024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
18842024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
18852024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
18862024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
18872024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
18882024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
18892024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
18902024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
18912024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
18922024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
18932024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
18942024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
18952024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
18962024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
18972024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
18982024-08-05T16:44:09.105ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
18992024-08-05T16:44:09.105ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 511083 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = f7f1e817-e56d-4529-8661-4bb1f7a7586e response_code = 500 uri = /omicron-zones
19002024-08-05T16:44:09.105ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f7f1e817-e56d-4529-8661-4bb1f7a7586e", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:09 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f7f1e817-e56d-4529-8661-4bb1f7a7586e" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.24030094s sled_address = [fd00:1122:3344:101::1]:12345
19012024-08-05T16:44:10.350ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
19022024-08-05T16:44:10.350ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19032024-08-05T16:44:10.350ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19042024-08-05T16:44:10.373ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19052024-08-05T16:44:10.373ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
19062024-08-05T16:44:10.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19072024-08-05T16:44:10.394ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19082024-08-05T16:44:10.415ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19092024-08-05T16:44:10.416ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19102024-08-05T16:44:10.435ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19112024-08-05T16:44:10.435ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19122024-08-05T16:44:10.453ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19132024-08-05T16:44:10.453ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19142024-08-05T16:44:10.473ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19152024-08-05T16:44:10.473ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19162024-08-05T16:44:10.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19172024-08-05T16:44:10.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19182024-08-05T16:44:10.511ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19192024-08-05T16:44:10.511ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19202024-08-05T16:44:10.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19212024-08-05T16:44:10.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19222024-08-05T16:44:10.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
19232024-08-05T16:44:10.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19242024-08-05T16:44:10.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19252024-08-05T16:44:10.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
19262024-08-05T16:44:10.596ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19272024-08-05T16:44:10.596ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
19282024-08-05T16:44:10.617ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19292024-08-05T16:44:10.617ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
19302024-08-05T16:44:10.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19312024-08-05T16:44:10.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
19322024-08-05T16:44:10.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19332024-08-05T16:44:10.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
19342024-08-05T16:44:10.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19352024-08-05T16:44:10.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
19362024-08-05T16:44:10.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19372024-08-05T16:44:10.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
19382024-08-05T16:44:10.720ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19392024-08-05T16:44:10.720ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
19402024-08-05T16:44:10.740ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19412024-08-05T16:44:10.740ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
19422024-08-05T16:44:10.762ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
19432024-08-05T16:44:10.762ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
19442024-08-05T16:44:10.762ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
19452024-08-05T16:44:10.762ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
19462024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
19472024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
19482024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
19492024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
19502024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
19512024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
19522024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
19532024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
19542024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
19552024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
19562024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
19572024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
19582024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
19592024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
19602024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
19612024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
19622024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
19632024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
19642024-08-05T16:44:10.866ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
19652024-08-05T16:44:10.866ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 514996 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = e56ab5a0-7acd-443a-9887-725b54131dce response_code = 500 uri = /omicron-zones
19662024-08-05T16:44:10.866ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e56ab5a0-7acd-443a-9887-725b54131dce", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:10 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e56ab5a0-7acd-443a-9887-725b54131dce" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.574239034s sled_address = [fd00:1122:3344:101::1]:12345
19672024-08-05T16:44:12.446ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
19682024-08-05T16:44:12.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19692024-08-05T16:44:12.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19702024-08-05T16:44:12.469ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19712024-08-05T16:44:12.469ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
19722024-08-05T16:44:12.488ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19732024-08-05T16:44:12.488ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19742024-08-05T16:44:12.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19752024-08-05T16:44:12.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19762024-08-05T16:44:12.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19772024-08-05T16:44:12.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19782024-08-05T16:44:12.546ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19792024-08-05T16:44:12.546ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19802024-08-05T16:44:12.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19812024-08-05T16:44:12.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19822024-08-05T16:44:12.583ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
19832024-08-05T16:44:12.583ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19842024-08-05T16:44:12.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19852024-08-05T16:44:12.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19862024-08-05T16:44:12.625ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
19872024-08-05T16:44:12.625ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19882024-08-05T16:44:12.646ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
19892024-08-05T16:44:12.646ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19902024-08-05T16:44:12.666ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19912024-08-05T16:44:12.666ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
19922024-08-05T16:44:12.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19932024-08-05T16:44:12.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
19942024-08-05T16:44:12.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19952024-08-05T16:44:12.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
19962024-08-05T16:44:12.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19972024-08-05T16:44:12.734ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
19982024-08-05T16:44:12.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
19992024-08-05T16:44:12.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
20002024-08-05T16:44:12.779ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20012024-08-05T16:44:12.779ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
20022024-08-05T16:44:12.801ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20032024-08-05T16:44:12.801ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
20042024-08-05T16:44:12.822ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20052024-08-05T16:44:12.822ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
20062024-08-05T16:44:12.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20072024-08-05T16:44:12.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
20082024-08-05T16:44:12.866ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
20092024-08-05T16:44:12.866ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
20102024-08-05T16:44:12.866ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
20112024-08-05T16:44:12.866ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
20122024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
20132024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
20142024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
20152024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
20162024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
20172024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
20182024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
20192024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
20202024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
20212024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
20222024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
20232024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
20242024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
20252024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
20262024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
20272024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
20282024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
20292024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
20302024-08-05T16:44:12.964ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
20312024-08-05T16:44:12.964ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 516005 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 3022681b-17a9-4f9e-a324-40a4ad9e5d7f response_code = 500 uri = /omicron-zones
20322024-08-05T16:44:12.964ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3022681b-17a9-4f9e-a324-40a4ad9e5d7f", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3022681b-17a9-4f9e-a324-40a4ad9e5d7f" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.988941976s sled_address = [fd00:1122:3344:101::1]:12345
20332024-08-05T16:44:14.961ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
20342024-08-05T16:44:14.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
20352024-08-05T16:44:14.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20362024-08-05T16:44:14.984ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
20372024-08-05T16:44:14.984ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
20382024-08-05T16:44:15.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
20392024-08-05T16:44:15.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20402024-08-05T16:44:15.026ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
20412024-08-05T16:44:15.026ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20422024-08-05T16:44:15.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
20432024-08-05T16:44:15.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20442024-08-05T16:44:15.067ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
20452024-08-05T16:44:15.067ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20462024-08-05T16:44:15.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
20472024-08-05T16:44:15.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20482024-08-05T16:44:15.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
20492024-08-05T16:44:15.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20502024-08-05T16:44:15.124ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
20512024-08-05T16:44:15.124ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20522024-08-05T16:44:15.143ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
20532024-08-05T16:44:15.143ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20542024-08-05T16:44:15.162ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
20552024-08-05T16:44:15.162ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20562024-08-05T16:44:15.180ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20572024-08-05T16:44:15.180ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
20582024-08-05T16:44:15.198ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20592024-08-05T16:44:15.198ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
20602024-08-05T16:44:15.217ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20612024-08-05T16:44:15.217ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
20622024-08-05T16:44:15.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20632024-08-05T16:44:15.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
20642024-08-05T16:44:15.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20652024-08-05T16:44:15.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
20662024-08-05T16:44:15.271ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20672024-08-05T16:44:15.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
20682024-08-05T16:44:15.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20692024-08-05T16:44:15.291ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
20702024-08-05T16:44:15.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20712024-08-05T16:44:15.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
20722024-08-05T16:44:15.328ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
20732024-08-05T16:44:15.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
20742024-08-05T16:44:15.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
20752024-08-05T16:44:15.348ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
20762024-08-05T16:44:15.348ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
20772024-08-05T16:44:15.348ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
20782024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
20792024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
20802024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
20812024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
20822024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
20832024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
20842024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
20852024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
20862024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
20872024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
20882024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
20892024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
20902024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
20912024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
20922024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
20932024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
20942024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
20952024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
20962024-08-05T16:44:15.440ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
20972024-08-05T16:44:15.441ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 478418 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 2feea260-6138-40dc-a7c8-f9e9088013ff response_code = 500 uri = /omicron-zones
20982024-08-05T16:44:15.441ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2feea260-6138-40dc-a7c8-f9e9088013ff", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2feea260-6138-40dc-a7c8-f9e9088013ff" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.952355604s sled_address = [fd00:1122:3344:101::1]:12345
20992024-08-05T16:44:17.400ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
21002024-08-05T16:44:17.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21012024-08-05T16:44:17.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21022024-08-05T16:44:17.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21032024-08-05T16:44:17.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
21042024-08-05T16:44:17.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21052024-08-05T16:44:17.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21062024-08-05T16:44:17.463ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21072024-08-05T16:44:17.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21082024-08-05T16:44:17.481ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21092024-08-05T16:44:17.481ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21102024-08-05T16:44:17.502ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21112024-08-05T16:44:17.502ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21122024-08-05T16:44:17.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21132024-08-05T16:44:17.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21142024-08-05T16:44:17.539ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21152024-08-05T16:44:17.539ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21162024-08-05T16:44:17.559ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21172024-08-05T16:44:17.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21182024-08-05T16:44:17.578ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21192024-08-05T16:44:17.578ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21202024-08-05T16:44:17.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
21212024-08-05T16:44:17.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21222024-08-05T16:44:17.616ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21232024-08-05T16:44:17.616ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
21242024-08-05T16:44:17.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21252024-08-05T16:44:17.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
21262024-08-05T16:44:17.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21272024-08-05T16:44:17.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
21282024-08-05T16:44:17.672ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21292024-08-05T16:44:17.672ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
21302024-08-05T16:44:17.691ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21312024-08-05T16:44:17.691ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
21322024-08-05T16:44:17.711ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21332024-08-05T16:44:17.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
21342024-08-05T16:44:17.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21352024-08-05T16:44:17.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
21362024-08-05T16:44:17.751ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21372024-08-05T16:44:17.751ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
21382024-08-05T16:44:17.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21392024-08-05T16:44:17.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
21402024-08-05T16:44:17.789ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
21412024-08-05T16:44:17.789ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
21422024-08-05T16:44:17.789ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
21432024-08-05T16:44:17.789ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
21442024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
21452024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
21462024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
21472024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
21482024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
21492024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
21502024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
21512024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
21522024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
21532024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
21542024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
21552024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
21562024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
21572024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
21582024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
21592024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
21602024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
21612024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
21622024-08-05T16:44:17.883ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
21632024-08-05T16:44:17.883ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 481423 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 392529e2-3544-4754-83b4-745867e623c5 response_code = 500 uri = /omicron-zones
21642024-08-05T16:44:17.883ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "392529e2-3544-4754-83b4-745867e623c5", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "392529e2-3544-4754-83b4-745867e623c5" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 2.412328217s sled_address = [fd00:1122:3344:101::1]:12345
21652024-08-05T16:44:20.303ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
21662024-08-05T16:44:20.304ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21672024-08-05T16:44:20.304ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21682024-08-05T16:44:20.327ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21692024-08-05T16:44:20.329ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
21702024-08-05T16:44:20.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21712024-08-05T16:44:20.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21722024-08-05T16:44:20.367ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21732024-08-05T16:44:20.367ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21742024-08-05T16:44:20.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21752024-08-05T16:44:20.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21762024-08-05T16:44:20.404ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21772024-08-05T16:44:20.404ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21782024-08-05T16:44:20.422ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21792024-08-05T16:44:20.422ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21802024-08-05T16:44:20.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
21812024-08-05T16:44:20.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21822024-08-05T16:44:20.460ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21832024-08-05T16:44:20.460ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21842024-08-05T16:44:20.479ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
21852024-08-05T16:44:20.479ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21862024-08-05T16:44:20.499ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
21872024-08-05T16:44:20.499ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21882024-08-05T16:44:20.518ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21892024-08-05T16:44:20.518ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
21902024-08-05T16:44:20.536ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21912024-08-05T16:44:20.536ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
21922024-08-05T16:44:20.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21932024-08-05T16:44:20.555ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
21942024-08-05T16:44:20.573ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21952024-08-05T16:44:20.573ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
21962024-08-05T16:44:20.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21972024-08-05T16:44:20.594ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
21982024-08-05T16:44:20.611ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
21992024-08-05T16:44:20.611ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
22002024-08-05T16:44:20.629ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22012024-08-05T16:44:20.629ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
22022024-08-05T16:44:20.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22032024-08-05T16:44:20.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
22042024-08-05T16:44:20.668ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22052024-08-05T16:44:20.668ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
22062024-08-05T16:44:20.686ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22072024-08-05T16:44:20.686ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
22082024-08-05T16:44:20.687ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
22092024-08-05T16:44:20.687ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22102024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
22112024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
22122024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
22132024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
22142024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
22152024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
22162024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
22172024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
22182024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
22192024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
22202024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
22212024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
22222024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
22232024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
22242024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
22252024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
22262024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
22272024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
22282024-08-05T16:44:20.780ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
22292024-08-05T16:44:20.780ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 475007 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = b6864bd3-95d2-40fe-98ba-083f9cc364f3 response_code = 500 uri = /omicron-zones
22302024-08-05T16:44:20.780ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b6864bd3-95d2-40fe-98ba-083f9cc364f3", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b6864bd3-95d2-40fe-98ba-083f9cc364f3" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 3.14396487s sled_address = [fd00:1122:3344:101::1]:12345
22312024-08-05T16:44:21.858ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.698608483s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
22322024-08-05T16:44:23.428ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60189
22332024-08-05T16:44:23.428ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:60189 req_id = f6fe5078-a6af-4cea-9744-90bd101ac1fd response_code = 200 uri = /baseboard
22342024-08-05T16:44:23.933ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
22352024-08-05T16:44:23.934ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
22362024-08-05T16:44:23.934ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22372024-08-05T16:44:23.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
22382024-08-05T16:44:23.957ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
22392024-08-05T16:44:23.978ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
22402024-08-05T16:44:23.978ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22412024-08-05T16:44:23.999ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
22422024-08-05T16:44:23.999ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22432024-08-05T16:44:24.022ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
22442024-08-05T16:44:24.022ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22452024-08-05T16:44:24.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
22462024-08-05T16:44:24.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22472024-08-05T16:44:24.061ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
22482024-08-05T16:44:24.061ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22492024-08-05T16:44:24.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
22502024-08-05T16:44:24.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22512024-08-05T16:44:24.099ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
22522024-08-05T16:44:24.099ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22532024-08-05T16:44:24.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
22542024-08-05T16:44:24.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22552024-08-05T16:44:24.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
22562024-08-05T16:44:24.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22572024-08-05T16:44:24.155ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22582024-08-05T16:44:24.155ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
22592024-08-05T16:44:24.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22602024-08-05T16:44:24.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
22612024-08-05T16:44:24.193ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22622024-08-05T16:44:24.193ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
22632024-08-05T16:44:24.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22642024-08-05T16:44:24.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
22652024-08-05T16:44:24.231ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22662024-08-05T16:44:24.231ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
22672024-08-05T16:44:24.249ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22682024-08-05T16:44:24.249ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
22692024-08-05T16:44:24.268ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22702024-08-05T16:44:24.268ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
22712024-08-05T16:44:24.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22722024-08-05T16:44:24.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
22732024-08-05T16:44:24.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
22742024-08-05T16:44:24.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
22752024-08-05T16:44:24.325ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22762024-08-05T16:44:24.325ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
22772024-08-05T16:44:24.326ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
22782024-08-05T16:44:24.326ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
22792024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
22802024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
22812024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
22822024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
22832024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
22842024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
22852024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
22862024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
22872024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
22882024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
22892024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
22902024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
22912024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
22922024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
22932024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
22942024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
22952024-08-05T16:44:24.418ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
22962024-08-05T16:44:24.419ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
22972024-08-05T16:44:24.419ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
22982024-08-05T16:44:24.419ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 483851 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 8390633c-d249-468f-8de5-9021d88b894f response_code = 500 uri = /omicron-zones
22992024-08-05T16:44:24.419ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8390633c-d249-468f-8de5-9021d88b894f", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:23 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8390633c-d249-468f-8de5-9021d88b894f" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 5.573290306s sled_address = [fd00:1122:3344:101::1]:12345
23002024-08-05T16:44:30.008ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
23012024-08-05T16:44:30.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23022024-08-05T16:44:30.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23032024-08-05T16:44:30.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23042024-08-05T16:44:30.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
23052024-08-05T16:44:30.049ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23062024-08-05T16:44:30.049ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23072024-08-05T16:44:30.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23082024-08-05T16:44:30.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23092024-08-05T16:44:30.090ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23102024-08-05T16:44:30.090ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23112024-08-05T16:44:30.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23122024-08-05T16:44:30.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23132024-08-05T16:44:30.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23142024-08-05T16:44:30.134ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23152024-08-05T16:44:30.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23162024-08-05T16:44:30.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23172024-08-05T16:44:30.171ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23182024-08-05T16:44:30.171ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23192024-08-05T16:44:30.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23202024-08-05T16:44:30.189ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23212024-08-05T16:44:30.206ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
23222024-08-05T16:44:30.206ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
23232024-08-05T16:44:30.225ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23242024-08-05T16:44:30.225ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
23252024-08-05T16:44:30.245ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23262024-08-05T16:44:30.245ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
23272024-08-05T16:44:30.263ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23282024-08-05T16:44:30.263ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
23292024-08-05T16:44:30.282ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23302024-08-05T16:44:30.282ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
23312024-08-05T16:44:30.300ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23322024-08-05T16:44:30.301ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
23332024-08-05T16:44:30.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23342024-08-05T16:44:30.319ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
23352024-08-05T16:44:30.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23362024-08-05T16:44:30.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
23372024-08-05T16:44:30.357ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23382024-08-05T16:44:30.357ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
23392024-08-05T16:44:30.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23402024-08-05T16:44:30.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
23412024-08-05T16:44:30.394ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
23422024-08-05T16:44:30.394ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
23432024-08-05T16:44:30.394ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
23442024-08-05T16:44:30.394ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
23452024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
23462024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
23472024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
23482024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
23492024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
23502024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
23512024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
23522024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
23532024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
23542024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
23552024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
23562024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
23572024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
23582024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
23592024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
23602024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
23612024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
23622024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
23632024-08-05T16:44:30.487ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
23642024-08-05T16:44:30.487ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 478188 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 4e409dc8-4724-4109-bdc2-ef1f9df367c4 response_code = 500 uri = /omicron-zones
23652024-08-05T16:44:30.488ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4e409dc8-4724-4109-bdc2-ef1f9df367c4", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4e409dc8-4724-4109-bdc2-ef1f9df367c4" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.64298186s sled_address = [fd00:1122:3344:101::1]:12345
23662024-08-05T16:44:35.144ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
23672024-08-05T16:44:35.144ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23682024-08-05T16:44:35.144ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23692024-08-05T16:44:35.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23702024-08-05T16:44:35.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
23712024-08-05T16:44:35.188ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23722024-08-05T16:44:35.188ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23732024-08-05T16:44:35.207ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23742024-08-05T16:44:35.207ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23752024-08-05T16:44:35.226ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23762024-08-05T16:44:35.227ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23772024-08-05T16:44:35.244ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23782024-08-05T16:44:35.244ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23792024-08-05T16:44:35.262ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23802024-08-05T16:44:35.262ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23812024-08-05T16:44:35.280ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
23822024-08-05T16:44:35.280ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23832024-08-05T16:44:35.298ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23842024-08-05T16:44:35.298ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23852024-08-05T16:44:35.316ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
23862024-08-05T16:44:35.317ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23872024-08-05T16:44:35.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
23882024-08-05T16:44:35.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
23892024-08-05T16:44:35.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23902024-08-05T16:44:35.353ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
23912024-08-05T16:44:35.371ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23922024-08-05T16:44:35.371ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
23932024-08-05T16:44:35.390ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23942024-08-05T16:44:35.390ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
23952024-08-05T16:44:35.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23962024-08-05T16:44:35.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
23972024-08-05T16:44:35.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
23982024-08-05T16:44:35.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
23992024-08-05T16:44:35.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24002024-08-05T16:44:35.445ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
24012024-08-05T16:44:35.465ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24022024-08-05T16:44:35.465ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
24032024-08-05T16:44:35.483ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24042024-08-05T16:44:35.483ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
24052024-08-05T16:44:35.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24062024-08-05T16:44:35.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
24072024-08-05T16:44:35.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
24082024-08-05T16:44:35.520ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
24092024-08-05T16:44:35.520ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
24102024-08-05T16:44:35.520ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
24112024-08-05T16:44:35.613ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
24122024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
24132024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
24142024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
24152024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
24162024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
24172024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
24182024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
24192024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
24202024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
24212024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
24222024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
24232024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
24242024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
24252024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
24262024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
24272024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
24282024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
24292024-08-05T16:44:35.614ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
24302024-08-05T16:44:35.614ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 468664 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 145103fd-4ba8-4039-a47d-33d5b5b2bfff response_code = 500 uri = /omicron-zones
24312024-08-05T16:44:35.614ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "145103fd-4ba8-4039-a47d-33d5b5b2bfff", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "145103fd-4ba8-4039-a47d-33d5b5b2bfff" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 9.864373544s sled_address = [fd00:1122:3344:101::1]:12345
24322024-08-05T16:44:40.644ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 10.16110074s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
24332024-08-05T16:44:45.506ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
24342024-08-05T16:44:45.506ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
24352024-08-05T16:44:45.506ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24362024-08-05T16:44:45.528ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
24372024-08-05T16:44:45.528ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
24382024-08-05T16:44:45.548ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
24392024-08-05T16:44:45.548ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24402024-08-05T16:44:45.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
24412024-08-05T16:44:45.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24422024-08-05T16:44:45.583ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
24432024-08-05T16:44:45.583ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24442024-08-05T16:44:45.601ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
24452024-08-05T16:44:45.602ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24462024-08-05T16:44:45.619ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
24472024-08-05T16:44:45.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24482024-08-05T16:44:45.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
24492024-08-05T16:44:45.641ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24502024-08-05T16:44:45.657ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
24512024-08-05T16:44:45.657ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24522024-08-05T16:44:45.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
24532024-08-05T16:44:45.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24542024-08-05T16:44:45.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
24552024-08-05T16:44:45.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24562024-08-05T16:44:45.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24572024-08-05T16:44:45.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
24582024-08-05T16:44:45.735ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24592024-08-05T16:44:45.735ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
24602024-08-05T16:44:45.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24612024-08-05T16:44:45.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
24622024-08-05T16:44:45.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24632024-08-05T16:44:45.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
24642024-08-05T16:44:45.790ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24652024-08-05T16:44:45.790ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
24662024-08-05T16:44:45.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24672024-08-05T16:44:45.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
24682024-08-05T16:44:45.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24692024-08-05T16:44:45.827ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
24702024-08-05T16:44:45.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24712024-08-05T16:44:45.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
24722024-08-05T16:44:45.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
24732024-08-05T16:44:45.865ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
24742024-08-05T16:44:45.882ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
24752024-08-05T16:44:45.882ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
24762024-08-05T16:44:45.882ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
24772024-08-05T16:44:45.882ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
24782024-08-05T16:44:45.976ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
24792024-08-05T16:44:45.976ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
24802024-08-05T16:44:45.976ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
24812024-08-05T16:44:45.976ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
24822024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
24832024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
24842024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
24852024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
24862024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
24872024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
24882024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
24892024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
24902024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
24912024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
24922024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
24932024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
24942024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
24952024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
24962024-08-05T16:44:45.977ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
24972024-08-05T16:44:45.977ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 469553 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 5e22d560-bc0c-4550-beea-c668930c2d24 response_code = 500 uri = /omicron-zones
24982024-08-05T16:44:45.977ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5e22d560-bc0c-4550-beea-c668930c2d24", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5e22d560-bc0c-4550-beea-c668930c2d24" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.900025189s sled_address = [fd00:1122:3344:101::1]:12345
24992024-08-05T16:44:50.892ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
25002024-08-05T16:44:50.892ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25012024-08-05T16:44:50.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25022024-08-05T16:44:50.914ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25032024-08-05T16:44:50.914ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
25042024-08-05T16:44:50.934ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25052024-08-05T16:44:50.934ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25062024-08-05T16:44:50.952ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25072024-08-05T16:44:50.952ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25082024-08-05T16:44:50.970ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25092024-08-05T16:44:50.970ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25102024-08-05T16:44:50.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25112024-08-05T16:44:50.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25122024-08-05T16:44:51.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25132024-08-05T16:44:51.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25142024-08-05T16:44:51.026ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25152024-08-05T16:44:51.026ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25162024-08-05T16:44:51.045ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25172024-08-05T16:44:51.045ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25182024-08-05T16:44:51.063ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25192024-08-05T16:44:51.063ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25202024-08-05T16:44:51.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
25212024-08-05T16:44:51.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
25222024-08-05T16:44:51.099ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25232024-08-05T16:44:51.099ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
25242024-08-05T16:44:51.117ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25252024-08-05T16:44:51.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
25262024-08-05T16:44:51.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25272024-08-05T16:44:51.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
25282024-08-05T16:44:51.156ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25292024-08-05T16:44:51.156ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
25302024-08-05T16:44:51.175ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25312024-08-05T16:44:51.175ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
25322024-08-05T16:44:51.194ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25332024-08-05T16:44:51.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
25342024-08-05T16:44:51.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25352024-08-05T16:44:51.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
25362024-08-05T16:44:51.231ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25372024-08-05T16:44:51.231ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
25382024-08-05T16:44:51.251ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25392024-08-05T16:44:51.251ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
25402024-08-05T16:44:51.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
25412024-08-05T16:44:51.270ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
25422024-08-05T16:44:51.271ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
25432024-08-05T16:44:51.271ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
25442024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
25452024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
25462024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
25472024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
25482024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
25492024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
25502024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
25512024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
25522024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
25532024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
25542024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
25552024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
25562024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
25572024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
25582024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
25592024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
25602024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
25612024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
25622024-08-05T16:44:51.364ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
25632024-08-05T16:44:51.365ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 471398 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = f69fccc9-7448-4256-8a45-c473dba8d6c7 response_code = 500 uri = /omicron-zones
25642024-08-05T16:44:51.365ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f69fccc9-7448-4256-8a45-c473dba8d6c7", "content-length": "124", "date": "Mon, 05 Aug 2024 16:44:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f69fccc9-7448-4256-8a45-c473dba8d6c7" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 11.122751636s sled_address = [fd00:1122:3344:101::1]:12345
25652024-08-05T16:44:53.541ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59039
25662024-08-05T16:44:53.542ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 709 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59039 req_id = b1436ad7-9e32-44a2-8c09-dce965842009 response_code = 200 uri = /baseboard
25672024-08-05T16:45:02.519ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
25682024-08-05T16:45:02.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25692024-08-05T16:45:02.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25702024-08-05T16:45:02.545ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25712024-08-05T16:45:02.545ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
25722024-08-05T16:45:02.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25732024-08-05T16:45:02.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25742024-08-05T16:45:02.588ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25752024-08-05T16:45:02.588ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25762024-08-05T16:45:02.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25772024-08-05T16:45:02.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25782024-08-05T16:45:02.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25792024-08-05T16:45:02.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25802024-08-05T16:45:02.645ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25812024-08-05T16:45:02.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25822024-08-05T16:45:02.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
25832024-08-05T16:45:02.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25842024-08-05T16:45:02.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25852024-08-05T16:45:02.681ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25862024-08-05T16:45:02.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
25872024-08-05T16:45:02.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25882024-08-05T16:45:02.719ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
25892024-08-05T16:45:02.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
25902024-08-05T16:45:02.739ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25912024-08-05T16:45:02.739ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
25922024-08-05T16:45:02.757ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25932024-08-05T16:45:02.757ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
25942024-08-05T16:45:02.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25952024-08-05T16:45:02.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
25962024-08-05T16:45:02.795ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25972024-08-05T16:45:02.795ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
25982024-08-05T16:45:02.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
25992024-08-05T16:45:02.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
26002024-08-05T16:45:02.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26012024-08-05T16:45:02.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
26022024-08-05T16:45:02.852ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26032024-08-05T16:45:02.852ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
26042024-08-05T16:45:02.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26052024-08-05T16:45:02.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
26062024-08-05T16:45:02.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26072024-08-05T16:45:02.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
26082024-08-05T16:45:02.908ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
26092024-08-05T16:45:02.908ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
26102024-08-05T16:45:02.908ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
26112024-08-05T16:45:02.909ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
26122024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
26132024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
26142024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
26152024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
26162024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
26172024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
26182024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
26192024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
26202024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
26212024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
26222024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
26232024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
26242024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
26252024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
26262024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
26272024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
26282024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
26292024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
26302024-08-05T16:45:03.005ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
26312024-08-05T16:45:03.005ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 483911 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = ec9da5dc-1abc-43f1-b868-784869382139 response_code = 500 uri = /omicron-zones
26322024-08-05T16:45:03.005ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ec9da5dc-1abc-43f1-b868-784869382139", "content-length": "124", "date": "Mon, 05 Aug 2024 16:45:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ec9da5dc-1abc-43f1-b868-784869382139" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 15.492124441s sled_address = [fd00:1122:3344:101::1]:12345
26332024-08-05T16:45:05.912ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 15.896631552s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
26342024-08-05T16:45:18.541ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
26352024-08-05T16:45:18.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
26362024-08-05T16:45:18.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26372024-08-05T16:45:18.563ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
26382024-08-05T16:45:18.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
26392024-08-05T16:45:18.583ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
26402024-08-05T16:45:18.583ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26412024-08-05T16:45:18.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
26422024-08-05T16:45:18.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26432024-08-05T16:45:18.622ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
26442024-08-05T16:45:18.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26452024-08-05T16:45:18.641ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
26462024-08-05T16:45:18.643ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26472024-08-05T16:45:18.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
26482024-08-05T16:45:18.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26492024-08-05T16:45:18.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
26502024-08-05T16:45:18.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26512024-08-05T16:45:18.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
26522024-08-05T16:45:18.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26532024-08-05T16:45:18.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
26542024-08-05T16:45:18.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26552024-08-05T16:45:18.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
26562024-08-05T16:45:18.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
26572024-08-05T16:45:18.756ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26582024-08-05T16:45:18.756ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
26592024-08-05T16:45:18.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26602024-08-05T16:45:18.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
26612024-08-05T16:45:18.794ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26622024-08-05T16:45:18.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
26632024-08-05T16:45:18.813ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26642024-08-05T16:45:18.813ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
26652024-08-05T16:45:18.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26662024-08-05T16:45:18.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
26672024-08-05T16:45:18.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26682024-08-05T16:45:18.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
26692024-08-05T16:45:18.869ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26702024-08-05T16:45:18.869ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
26712024-08-05T16:45:18.887ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26722024-08-05T16:45:18.887ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
26732024-08-05T16:45:18.906ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
26742024-08-05T16:45:18.906ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
26752024-08-05T16:45:18.925ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
26762024-08-05T16:45:18.925ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
26772024-08-05T16:45:18.925ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
26782024-08-05T16:45:18.925ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
26792024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
26802024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
26812024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
26822024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
26832024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
26842024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
26852024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
26862024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
26872024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
26882024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
26892024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
26902024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
26912024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
26922024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
26932024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
26942024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
26952024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
26962024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
26972024-08-05T16:45:19.020ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
26982024-08-05T16:45:19.020ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 477456 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = aea4e807-bd57-4097-8608-4714b04e8d5a response_code = 500 uri = /omicron-zones
26992024-08-05T16:45:19.020ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "aea4e807-bd57-4097-8608-4714b04e8d5a", "content-length": "124", "date": "Mon, 05 Aug 2024 16:45:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "aea4e807-bd57-4097-8608-4714b04e8d5a" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 14.035870625s sled_address = [fd00:1122:3344:101::1]:12345
27002024-08-05T16:45:23.660ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36117
27012024-08-05T16:45:23.660ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 41 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36117 req_id = 1d28d696-6516-4982-b260-a6c50a59cba2 response_code = 200 uri = /baseboard
27022024-08-05T16:45:33.095ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
27032024-08-05T16:45:33.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27042024-08-05T16:45:33.095ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27052024-08-05T16:45:33.116ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27062024-08-05T16:45:33.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
27072024-08-05T16:45:33.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27082024-08-05T16:45:33.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27092024-08-05T16:45:33.155ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27102024-08-05T16:45:33.155ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27112024-08-05T16:45:33.173ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27122024-08-05T16:45:33.175ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27132024-08-05T16:45:33.192ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27142024-08-05T16:45:33.192ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27152024-08-05T16:45:33.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27162024-08-05T16:45:33.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27172024-08-05T16:45:33.228ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27182024-08-05T16:45:33.229ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27192024-08-05T16:45:33.247ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27202024-08-05T16:45:33.247ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27212024-08-05T16:45:33.264ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27222024-08-05T16:45:33.264ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27232024-08-05T16:45:33.281ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
27242024-08-05T16:45:33.281ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
27252024-08-05T16:45:33.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27262024-08-05T16:45:33.305ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
27272024-08-05T16:45:33.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27282024-08-05T16:45:33.321ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
27292024-08-05T16:45:33.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27302024-08-05T16:45:33.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
27312024-08-05T16:45:33.358ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27322024-08-05T16:45:33.358ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
27332024-08-05T16:45:33.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27342024-08-05T16:45:33.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
27352024-08-05T16:45:33.394ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27362024-08-05T16:45:33.395ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
27372024-08-05T16:45:33.411ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27382024-08-05T16:45:33.411ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
27392024-08-05T16:45:33.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27402024-08-05T16:45:33.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
27412024-08-05T16:45:33.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27422024-08-05T16:45:33.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
27432024-08-05T16:45:33.465ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
27442024-08-05T16:45:33.465ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
27452024-08-05T16:45:33.465ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
27462024-08-05T16:45:33.465ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
27472024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
27482024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
27492024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
27502024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
27512024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
27522024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
27532024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
27542024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
27552024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
27562024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
27572024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
27582024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
27592024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
27602024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
27612024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
27622024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
27632024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
27642024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
27652024-08-05T16:45:33.557ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3065 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
27662024-08-05T16:45:33.558ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65", TimeNotSynchronized), ("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf", TimeNotSynchronized), ("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42", TimeNotSynchronized), ("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d", TimeNotSynchronized), ("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015", TimeNotSynchronized), ("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb", TimeNotSynchronized), ("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631", TimeNotSynchronized), ("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83", TimeNotSynchronized), ("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2", TimeNotSynchronized), ("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d", TimeNotSynchronized), ("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069", TimeNotSynchronized), ("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451", TimeNotSynchronized), ("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7", TimeNotSynchronized), ("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300", TimeNotSynchronized), ("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da", TimeNotSynchronized), ("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8", TimeNotSynchronized), ("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8", TimeNotSynchronized), ("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1", TimeNotSynchronized), ("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:902 latency_us = 461051 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 69737ee5-e729-47d5-b9a3-86fd5759d03f response_code = 500 uri = /omicron-zones
27672024-08-05T16:45:33.558ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "69737ee5-e729-47d5-b9a3-86fd5759d03f", "content-length": "124", "date": "Mon, 05 Aug 2024 16:45:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "69737ee5-e729-47d5-b9a3-86fd5759d03f" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 15.380709088s sled_address = [fd00:1122:3344:101::1]:12345
27682024-08-05T16:45:36.932ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 46.079444441s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
27692024-08-05T16:45:48.981ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 83a194e2-3464-4a84-ae47-e33a0983bc56, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: fe17df80-010a-4b68-9ebc-6a94437be875, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 6300d678-c957-4259-9a3e-c167ee4d0fb0, kind: Service { id: 5b6cee62-004c-4843-9e16-c25d2b0c743b }, name: Name("ntp-5b6cee62-004c-4843-9e16-c25d2b0c743b"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 71])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3c470f26-8422-4139-9f7f-81a0d84becba, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: acae5a41-5da9-420f-9bc8-154e4abdaf87, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: eed4ad2f-7765-43e1-8943-ea458ed22e57, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 902ea001-1913-4130-9979-87a8d3db1f53, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8995285b-5cb0-4d3a-b021-8aec05314a65, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 866d6328-4ca1-435e-8a98-cd5268df29e8, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 893afdd2-39c6-4f94-b381-facb1a40bbb1, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 061f99f8-6626-4a81-8262-abd77ceeb36a, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 15711896-d3e4-44e3-b330-624f0f5e064d, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: c02f008f-19c7-48c5-93c6-3f776c51e015, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 738351b7-dacb-486a-ab38-cf8aefaef9cb, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 29db7ccb-90e1-414b-aeee-a9393f639631, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External } } } }, OmicronZoneConfig { id: d857801a-a95b-4937-a2f0-31ad7c239ac7, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External } } } }, OmicronZoneConfig { id: 58b140e5-f61e-4410-8851-af929dfd8069, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External } } } }, OmicronZoneConfig { id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External } } } }, OmicronZoneConfig { id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External } } } }, OmicronZoneConfig { id: c1741444-7ab7-4888-aa3c-e7b6da3527da, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
27702024-08-05T16:45:48.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27712024-08-05T16:45:48.982ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 83a194e2-3464-4a84-ae47-e33a0983bc56, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27722024-08-05T16:45:49.004ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27732024-08-05T16:45:49.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 5145c3cc-86a8-4ca7-a5dd-d58654518f19, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: InternalDns }, responder: ... }
27742024-08-05T16:45:49.025ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27752024-08-05T16:45:49.025ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: fe17df80-010a-4b68-9ebc-6a94437be875, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27762024-08-05T16:45:49.044ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27772024-08-05T16:45:49.044ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 3c470f26-8422-4139-9f7f-81a0d84becba, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27782024-08-05T16:45:49.064ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27792024-08-05T16:45:49.064ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: acae5a41-5da9-420f-9bc8-154e4abdaf87, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27802024-08-05T16:45:49.083ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27812024-08-05T16:45:49.083ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: eed4ad2f-7765-43e1-8943-ea458ed22e57, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27822024-08-05T16:45:49.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27832024-08-05T16:45:49.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: b2f11b30-44fe-4b76-98c0-1891b1b0971a, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27842024-08-05T16:45:49.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:405
27852024-08-05T16:45:49.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 902ea001-1913-4130-9979-87a8d3db1f53, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27862024-08-05T16:45:49.139ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27872024-08-05T16:45:49.139ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8995285b-5cb0-4d3a-b021-8aec05314a65, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27882024-08-05T16:45:49.158ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:405
27892024-08-05T16:45:49.158ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27902024-08-05T16:45:49.178ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:405
27912024-08-05T16:45:49.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 061f99f8-6626-4a81-8262-abd77ceeb36a, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
27922024-08-05T16:45:49.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27932024-08-05T16:45:49.199ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 29db7ccb-90e1-414b-aeee-a9393f639631, dataset_name: DatasetName { pool_name: ZpoolName { id: bd033efe-47fe-4ba2-8bc4-ba99e628ada7 (zpool), kind: External }, kind: Crucible }, responder: ... }
27942024-08-05T16:45:49.217ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27952024-08-05T16:45:49.217ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 8029adce-2a1d-48e9-a435-7bcff4f3bb83, dataset_name: DatasetName { pool_name: ZpoolName { id: 1aa678c2-e32d-4428-a115-7e698777172b (zpool), kind: External }, kind: Crucible }, responder: ... }
27962024-08-05T16:45:49.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27972024-08-05T16:45:49.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: d857801a-a95b-4937-a2f0-31ad7c239ac7, dataset_name: DatasetName { pool_name: ZpoolName { id: cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396 (zpool), kind: External }, kind: Crucible }, responder: ... }
27982024-08-05T16:45:49.255ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
27992024-08-05T16:45:49.255ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 0d925b8a-ac40-4a91-9334-d6f7d7a29fb2, dataset_name: DatasetName { pool_name: ZpoolName { id: 939962e4-70b1-47be-b2d6-938c818b62c4 (zpool), kind: External }, kind: Crucible }, responder: ... }
28002024-08-05T16:45:49.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
28012024-08-05T16:45:49.275ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 48f67680-6ea6-4e0c-9968-c45baf393c1d, dataset_name: DatasetName { pool_name: ZpoolName { id: 52c33af8-e5fc-4cb6-93db-7a529f23be77 (zpool), kind: External }, kind: Crucible }, responder: ... }
28022024-08-05T16:45:49.293ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
28032024-08-05T16:45:49.293ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 58b140e5-f61e-4410-8851-af929dfd8069, dataset_name: DatasetName { pool_name: ZpoolName { id: cb14544a-cc03-4a04-84a7-da1d51f0a71d (zpool), kind: External }, kind: Crucible }, responder: ... }
28042024-08-05T16:45:49.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
28052024-08-05T16:45:49.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: f5fbccc2-b8b1-4dd2-9655-42d226ad2451, dataset_name: DatasetName { pool_name: ZpoolName { id: 2cb76428-77cf-4416-ab3e-f66fc49a03b4 (zpool), kind: External }, kind: Crucible }, responder: ... }
28062024-08-05T16:45:49.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
28072024-08-05T16:45:49.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: 19f0f364-8840-4b44-9eea-e3a6f21b6300, dataset_name: DatasetName { pool_name: ZpoolName { id: 1a42fe8d-3436-46f3-8051-0d2ff2a0be4f (zpool), kind: External }, kind: Crucible }, responder: ... }
28082024-08-05T16:45:49.350ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:405
28092024-08-05T16:45:49.350ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:772 request = NewFilesystemRequest { dataset_id: c1741444-7ab7-4888-aa3c-e7b6da3527da, dataset_name: DatasetName { pool_name: ZpoolName { id: 85b774a7-53a2-4f22-a027-1f642ba2d7f9 (zpool), kind: External }, kind: Crucible }, responder: ... }
28102024-08-05T16:45:49.369ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28112024-08-05T16:45:49.369ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
28122024-08-05T16:45:49.369ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
28132024-08-05T16:45:49.369ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28142024-08-05T16:45:49.460ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
28152024-08-05T16:45:49.475ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
28162024-08-05T16:45:49.475ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28172024-08-05T16:45:49.475ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28182024-08-05T16:45:49.486ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
28192024-08-05T16:45:49.545ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28202024-08-05T16:45:49.545ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28212024-08-05T16:45:49.545ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28222024-08-05T16:45:49.545ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28232024-08-05T16:45:49.545ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28242024-08-05T16:45:49.718ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
28252024-08-05T16:45:49.719ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28262024-08-05T16:45:49.719ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28272024-08-05T16:45:49.719ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28282024-08-05T16:45:49.719ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28292024-08-05T16:45:49.743ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
28302024-08-05T16:45:49.743ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28312024-08-05T16:45:49.743ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28322024-08-05T16:45:49.745ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28332024-08-05T16:45:49.745ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1 file = illumos-utils/src/zone.rs:316
28342024-08-05T16:45:49.750ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015 file = illumos-utils/src/zone.rs:316
28352024-08-05T16:45:49.754ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a file = illumos-utils/src/zone.rs:316
28362024-08-05T16:45:49.759ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb file = illumos-utils/src/zone.rs:316
28372024-08-05T16:45:49.764ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631 file = illumos-utils/src/zone.rs:316
28382024-08-05T16:45:49.769ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83 file = illumos-utils/src/zone.rs:316
28392024-08-05T16:45:49.773ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d file = illumos-utils/src/zone.rs:316
28402024-08-05T16:45:49.912ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1 file = illumos-utils/src/zone.rs:354
28412024-08-05T16:45:49.917ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015 file = illumos-utils/src/zone.rs:354
28422024-08-05T16:45:49.922ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a file = illumos-utils/src/zone.rs:354
28432024-08-05T16:45:49.926ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb file = illumos-utils/src/zone.rs:354
28442024-08-05T16:45:49.931ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631 file = illumos-utils/src/zone.rs:354
28452024-08-05T16:45:49.935ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83 file = illumos-utils/src/zone.rs:354
28462024-08-05T16:45:49.940ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d file = illumos-utils/src/zone.rs:354
28472024-08-05T16:45:49.945ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069 file = illumos-utils/src/zone.rs:316
28482024-08-05T16:45:49.952ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451 file = illumos-utils/src/zone.rs:316
28492024-08-05T16:45:49.957ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300 file = illumos-utils/src/zone.rs:316
28502024-08-05T16:45:49.963ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da file = illumos-utils/src/zone.rs:316
28512024-08-05T16:45:49.969ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7 file = illumos-utils/src/zone.rs:316
28522024-08-05T16:45:49.974ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2 file = illumos-utils/src/zone.rs:316
28532024-08-05T16:45:49.981ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d file = illumos-utils/src/zone.rs:316
28542024-08-05T16:45:50.024ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
28552024-08-05T16:45:50.024ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069 file = illumos-utils/src/zone.rs:354
28562024-08-05T16:45:50.031ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451 file = illumos-utils/src/zone.rs:354
28572024-08-05T16:45:50.038ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300 file = illumos-utils/src/zone.rs:354
28582024-08-05T16:45:50.044ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da file = illumos-utils/src/zone.rs:354
28592024-08-05T16:45:50.052ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7 file = illumos-utils/src/zone.rs:354
28602024-08-05T16:45:50.059ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2 file = illumos-utils/src/zone.rs:354
28612024-08-05T16:45:50.067ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d file = illumos-utils/src/zone.rs:354
28622024-08-05T16:45:50.217ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
28632024-08-05T16:45:50.293ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
28642024-08-05T16:45:50.297ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
28652024-08-05T16:45:50.297ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
28662024-08-05T16:45:50.351ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 202, 28]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
28672024-08-05T16:45:50.351ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = nexus
28682024-08-05T16:45:50.405ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
28692024-08-05T16:45:50.486ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
28702024-08-05T16:45:50.492ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
28712024-08-05T16:45:50.492ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
28722024-08-05T16:45:50.569ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 233, 199]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
28732024-08-05T16:45:50.569ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = external_dns
28742024-08-05T16:45:50.569ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
28752024-08-05T16:45:50.569ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
28762024-08-05T16:45:50.569ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
28772024-08-05T16:45:50.569ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
28782024-08-05T16:45:50.627ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 206, 204]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
28792024-08-05T16:45:50.628ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = external_dns
28802024-08-05T16:45:50.630ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
28812024-08-05T16:45:50.630ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
28822024-08-05T16:45:50.680ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 210, 250]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
28832024-08-05T16:45:50.680ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = nexus
28842024-08-05T16:45:50.682ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
28852024-08-05T16:45:50.682ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
28862024-08-05T16:45:50.731ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 216, 188]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
28872024-08-05T16:45:50.731ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1210 zone_type = nexus
28882024-08-05T16:45:50.731ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
28892024-08-05T16:45:50.733ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28902024-08-05T16:45:50.816ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
28912024-08-05T16:45:50.816ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
28922024-08-05T16:45:50.816ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
28932024-08-05T16:45:50.816ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
28942024-08-05T16:45:50.817ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
28952024-08-05T16:45:50.817ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
28962024-08-05T16:45:50.818ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
28972024-08-05T16:45:50.821ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
28982024-08-05T16:45:50.821ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
28992024-08-05T16:45:50.826ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
29002024-08-05T16:45:50.830ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
29012024-08-05T16:45:50.830ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
29022024-08-05T16:45:50.830ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
29032024-08-05T16:45:50.840ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
29042024-08-05T16:45:50.884ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 file = illumos-utils/src/zone.rs:316
29052024-08-05T16:45:50.976ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
29062024-08-05T16:45:50.978ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf file = illumos-utils/src/zone.rs:316
29072024-08-05T16:45:50.998ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
29082024-08-05T16:45:50.999ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 file = illumos-utils/src/zone.rs:354
29092024-08-05T16:45:51.012ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65 file = illumos-utils/src/zone.rs:316
29102024-08-05T16:45:51.027ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8 file = illumos-utils/src/zone.rs:316
29112024-08-05T16:45:51.227ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf file = illumos-utils/src/zone.rs:354
29122024-08-05T16:45:51.252ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65 file = illumos-utils/src/zone.rs:354
29132024-08-05T16:45:51.270ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8 file = illumos-utils/src/zone.rs:354
29142024-08-05T16:45:51.294ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8 file = illumos-utils/src/zone.rs:316
29152024-08-05T16:45:51.350ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8 file = illumos-utils/src/zone.rs:354
29162024-08-05T16:45:53.096ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29172024-08-05T16:45:53.096ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
29182024-08-05T16:45:53.250ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29192024-08-05T16:45:53.251ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
29202024-08-05T16:45:53.443ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_bd033efe-47fe-4ba2-8bc4-ba99e628ada7/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='310306f8-7d0f-4e6c-b569-abcf1e5494f4'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29212024-08-05T16:45:53.443ZINFOSledAgent (ServiceManager): Booting oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
29222024-08-05T16:45:53.453ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29232024-08-05T16:45:53.453ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
29242024-08-05T16:45:53.520ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='bf16fff1-899a-49e8-a740-a5a93aa70f8b'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29252024-08-05T16:45:53.520ZINFOSledAgent (ServiceManager): Booting oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
29262024-08-05T16:45:53.536ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_1aa678c2-e32d-4428-a115-7e698777172b/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='6a3f19bd-1d3b-4ef0-8fa3-0de6225526a6'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29272024-08-05T16:45:53.536ZINFOSledAgent (ServiceManager): Booting oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
29282024-08-05T16:45:53.569ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_cb14544a-cc03-4a04-84a7-da1d51f0a71d/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='265827d3-d2f1-4ea0-9002-f3c5ffe8c523'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29292024-08-05T16:45:53.571ZINFOSledAgent (ServiceManager): Booting oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
29302024-08-05T16:45:53.597ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_52c33af8-e5fc-4cb6-93db-7a529f23be77/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='1f219295-24a9-4e37-a221-d16c652f3654'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29312024-08-05T16:45:53.601ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='893afdd2-39c6-4f94-b381-facb1a40bbb1'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29322024-08-05T16:45:53.602ZINFOSledAgent (ServiceManager): Booting oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
29332024-08-05T16:45:53.633ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_939962e4-70b1-47be-b2d6-938c818b62c4/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='ade05617-a412-4501-a24c-019271cd3ac2'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29342024-08-05T16:45:53.639ZINFOSledAgent (ServiceManager): Booting oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
29352024-08-05T16:45:53.660ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
29362024-08-05T16:45:54.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 54.049664ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
29372024-08-05T16:45:54.932ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_85b774a7-53a2-4f22-a027-1f642ba2d7f9/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a743cd31-4e2e-4b0a-adc7-2436a82e05dc'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29382024-08-05T16:45:54.934ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_1a42fe8d-3436-46f3-8051-0d2ff2a0be4f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a302ac3c-8a0e-475d-a226-e072c8feb8d1'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29392024-08-05T16:45:54.935ZINFOSledAgent (ServiceManager): Booting oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
29402024-08-05T16:45:59.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 29.995804ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
29412024-08-05T16:46:00.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 102.725107ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
29422024-08-05T16:46:00.190ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_2cb76428-77cf-4416-ab3e-f66fc49a03b4/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='c1416d9c-403f-410d-84bc-b0d7f1cb1644'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
29432024-08-05T16:46:00.191ZINFOSledAgent (ServiceManager): Booting oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
29442024-08-05T16:46:00.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 68.059496ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
29452024-08-05T16:46:00.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 85.269964ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
29462024-08-05T16:46:00.722ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 223.644782ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
29472024-08-05T16:46:00.906ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37841
29482024-08-05T16:46:00.908ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1201 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37841 req_id = 3150380f-9718-4bfb-9c2b-8939b99047d0 response_code = 200 uri = /baseboard
29492024-08-05T16:46:00.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 32.246802ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
29502024-08-05T16:46:00.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 63.387478ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
29512024-08-05T16:46:01.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 53.76981ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
29522024-08-05T16:46:01.063ZINFOSledAgent (ServiceManager): Booting oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
29532024-08-05T16:46:01.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 44.93725ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
29542024-08-05T16:46:01.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 109.365207ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
29552024-08-05T16:46:01.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 250.213069ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
29562024-08-05T16:46:01.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 440.723297ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
29572024-08-05T16:46:01.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 99.052458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
29582024-08-05T16:46:01.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 104.954736ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
29592024-08-05T16:46:01.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 55.704472ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
29602024-08-05T16:46:01.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 105.429152ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
29612024-08-05T16:46:01.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 72.686803ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
29622024-08-05T16:46:01.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 87.236334ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
29632024-08-05T16:46:01.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 52.785714ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
29642024-08-05T16:46:02.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 238.346862ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
29652024-08-05T16:46:02.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 35.519733ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
29662024-08-05T16:46:02.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 236.267927ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
29672024-08-05T16:46:02.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 563.669882ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
29682024-08-05T16:46:03.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 272.444074ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
29692024-08-05T16:46:03.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 247.513277ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
29702024-08-05T16:46:03.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 81.433634ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
29712024-08-05T16:46:03.284ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 113.634146ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
29722024-08-05T16:46:03.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 54.188475ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
29732024-08-05T16:46:03.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 47.593918ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
29742024-08-05T16:46:03.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 63.700525ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
29752024-08-05T16:46:03.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 296.585249ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
29762024-08-05T16:46:03.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 142.052806ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
29772024-08-05T16:46:03.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 405.330203ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
29782024-08-05T16:46:03.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 89.503664ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
29792024-08-05T16:46:03.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 665.414831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
29802024-08-05T16:46:03.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 185.974552ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
29812024-08-05T16:46:03.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 337.070339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
29822024-08-05T16:46:03.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 144.65644ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
29832024-08-05T16:46:03.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 77.662037ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
29842024-08-05T16:46:03.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 284.920688ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
29852024-08-05T16:46:03.502ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 831.960904ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
29862024-08-05T16:46:03.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 298.068101ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
29872024-08-05T16:46:03.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 238.276856ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
29882024-08-05T16:46:03.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 416.306213ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
29892024-08-05T16:46:03.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 183.161355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
29902024-08-05T16:46:03.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 265.838338ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
29912024-08-05T16:46:03.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 262.661049ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
29922024-08-05T16:46:03.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 229.975198ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
29932024-08-05T16:46:03.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 418.73667ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
29942024-08-05T16:46:03.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 581.669305ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
29952024-08-05T16:46:03.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 254.709467ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
29962024-08-05T16:46:03.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 790.715713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
29972024-08-05T16:46:03.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 553.128675ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
29982024-08-05T16:46:03.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 547.697325ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
29992024-08-05T16:46:03.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 661.98814ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
30002024-08-05T16:46:03.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 239.934223ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
30012024-08-05T16:46:03.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 678.281081ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
30022024-08-05T16:46:03.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 346.257073ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
30032024-08-05T16:46:03.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 662.981339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
30042024-08-05T16:46:04.009ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30052024-08-05T16:46:04.009ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30062024-08-05T16:46:04.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 975.960684ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
30072024-08-05T16:46:04.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 469.807383ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
30082024-08-05T16:46:04.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.216768402s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
30092024-08-05T16:46:04.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 1.074222781s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
30102024-08-05T16:46:04.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 589.722914ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
30112024-08-05T16:46:04.278ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 27.575233ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30122024-08-05T16:46:04.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 64.304588ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30132024-08-05T16:46:04.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 1.406746961s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
30142024-08-05T16:46:04.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 842.76938ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
30152024-08-05T16:46:04.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.391522665s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
30162024-08-05T16:46:04.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 660.417478ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
30172024-08-05T16:46:04.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 244.106689ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30182024-08-05T16:46:04.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 641.405929ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
30192024-08-05T16:46:04.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 754.67835ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
30202024-08-05T16:46:04.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 1.051342392s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
30212024-08-05T16:46:04.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 1.009569103s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
30222024-08-05T16:46:04.662ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 1.006487994s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
30232024-08-05T16:46:04.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 565.9474ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30242024-08-05T16:46:04.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.362384841s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
30252024-08-05T16:46:05.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 1.212943843s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
30262024-08-05T16:46:05.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.396750098s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
30272024-08-05T16:46:05.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 919.159478ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
30282024-08-05T16:46:05.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 617.142838ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
30292024-08-05T16:46:05.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.431828929s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
30302024-08-05T16:46:05.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 585.662782ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30312024-08-05T16:46:05.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 837.405844ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
30322024-08-05T16:46:05.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 979.603639ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
30332024-08-05T16:46:05.638ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30342024-08-05T16:46:05.638ZINFOSledAgent (ServiceManager): Booting oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
30352024-08-05T16:46:05.645ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30362024-08-05T16:46:05.646ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30372024-08-05T16:46:05.646ZINFOSledAgent (ServiceManager): Booting oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
30382024-08-05T16:46:05.662ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 1.426962537s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
30392024-08-05T16:46:05.663ZINFOSledAgent (ServiceManager): Booting oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
30402024-08-05T16:46:05.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 992.133089ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
30412024-08-05T16:46:05.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 1.162271041s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
30422024-08-05T16:46:05.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 864.16091ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
30432024-08-05T16:46:05.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.246929844s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
30442024-08-05T16:46:05.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 809.89581ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
30452024-08-05T16:46:05.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 61.764374ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
30462024-08-05T16:46:05.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 29.804225ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
30472024-08-05T16:46:05.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 71.910708ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
30482024-08-05T16:46:05.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 755.528258ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30492024-08-05T16:46:05.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 147.769337ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
30502024-08-05T16:46:05.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 102.007386ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
30512024-08-05T16:46:05.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 138.051814ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
30522024-08-05T16:46:06.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 287.192137ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
30532024-08-05T16:46:06.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 119.069442ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
30542024-08-05T16:46:06.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.113158796s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
30552024-08-05T16:46:06.130ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 196.354594ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
30562024-08-05T16:46:06.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 593.156118ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
30572024-08-05T16:46:06.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 249.415189ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
30582024-08-05T16:46:06.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 771.308838ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
30592024-08-05T16:46:06.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 595.518301ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
30602024-08-05T16:46:06.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.172735292s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
30612024-08-05T16:46:06.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 749.45941ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30622024-08-05T16:46:06.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 987.709553ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
30632024-08-05T16:46:06.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 529.807254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
30642024-08-05T16:46:06.769ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 906.251316ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
30652024-08-05T16:46:06.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 538.434642ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
30662024-08-05T16:46:06.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 599.13461ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
30672024-08-05T16:46:06.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 1.24121913s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
30682024-08-05T16:46:06.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 818.530437ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
30692024-08-05T16:46:06.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 705.701305ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
30702024-08-05T16:46:06.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 995.001332ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
30712024-08-05T16:46:06.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 904.101238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
30722024-08-05T16:46:07.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 962.233139ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
30732024-08-05T16:46:07.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 701.616375ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
30742024-08-05T16:46:07.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 777.935068ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
30752024-08-05T16:46:07.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 1.0537227s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
30762024-08-05T16:46:07.284ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.457969697s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
30772024-08-05T16:46:07.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 699.93187ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
30782024-08-05T16:46:07.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 1.427347132s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
30792024-08-05T16:46:07.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.069317264s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
30802024-08-05T16:46:07.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 946.060841ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30812024-08-05T16:46:07.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.160570434s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
30822024-08-05T16:46:07.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 531.074992ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
30832024-08-05T16:46:07.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 742.521119ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
30842024-08-05T16:46:07.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 900.796042ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
30852024-08-05T16:46:08.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 619.747711ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
30862024-08-05T16:46:08.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 874.291676ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
30872024-08-05T16:46:08.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 713.435835ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
30882024-08-05T16:46:08.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.098490243s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
30892024-08-05T16:46:08.238ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 562.030555ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
30902024-08-05T16:46:08.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 707.013402ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
30912024-08-05T16:46:08.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.461419908s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
30922024-08-05T16:46:08.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.195002182s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
30932024-08-05T16:46:08.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.293793382s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
30942024-08-05T16:46:08.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 879.320129ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
30952024-08-05T16:46:08.488ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 717.771322ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
30962024-08-05T16:46:08.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.32602712s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
30972024-08-05T16:46:08.497ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30982024-08-05T16:46:08.498ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
30992024-08-05T16:46:08.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.082215582s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
31002024-08-05T16:46:08.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 1.351057199s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
31012024-08-05T16:46:08.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 809.159015ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
31022024-08-05T16:46:08.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 719.950018ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
31032024-08-05T16:46:08.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 623.827575ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
31042024-08-05T16:46:08.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 1.082125757s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
31052024-08-05T16:46:08.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 1.453831759s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
31062024-08-05T16:46:08.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 56.978128ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
31072024-08-05T16:46:08.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 110.714032ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
31082024-08-05T16:46:08.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 658.574027ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
31092024-08-05T16:46:08.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 537.073937ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
31102024-08-05T16:46:09.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 295.714363ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
31112024-08-05T16:46:09.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 1.027623289s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
31122024-08-05T16:46:09.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 959.61914ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
31132024-08-05T16:46:09.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.360645786s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
31142024-08-05T16:46:09.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 258.903855ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
31152024-08-05T16:46:09.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 879.969348ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
31162024-08-05T16:46:09.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 637.440379ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
31172024-08-05T16:46:09.469ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.081082694s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
31182024-08-05T16:46:09.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.151174811s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
31192024-08-05T16:46:09.511ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.404238864s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
31202024-08-05T16:46:09.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.289180666s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
31212024-08-05T16:46:09.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.235261758s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
31222024-08-05T16:46:09.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 549.648292ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
31232024-08-05T16:46:09.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 951.858985ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
31242024-08-05T16:46:09.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 973.413074ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
31252024-08-05T16:46:09.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.052676908s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
31262024-08-05T16:46:09.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.106320683s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
31272024-08-05T16:46:09.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 1.263266409s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
31282024-08-05T16:46:10.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 665.535389ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
31292024-08-05T16:46:10.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 890.9809ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
31302024-08-05T16:46:10.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 679.833069ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
31312024-08-05T16:46:10.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 936.041921ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
31322024-08-05T16:46:10.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 1.464961809s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
31332024-08-05T16:46:10.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 1.158379503s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
31342024-08-05T16:46:10.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 529.036754ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
31352024-08-05T16:46:10.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 983.15834ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
31362024-08-05T16:46:10.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 823.657352ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
31372024-08-05T16:46:10.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 829.828817ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
31382024-08-05T16:46:10.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 958.648153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
31392024-08-05T16:46:10.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 811.774345ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
31402024-08-05T16:46:10.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 960.407462ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
31412024-08-05T16:46:10.787ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 767.16051ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
31422024-08-05T16:46:10.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.215645852s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
31432024-08-05T16:46:10.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 1.286734092s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
31442024-08-05T16:46:10.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 570.130839ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
31452024-08-05T16:46:10.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 766.823499ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
31462024-08-05T16:46:11.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.159893481s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
31472024-08-05T16:46:11.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 844.825453ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
31482024-08-05T16:46:11.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 726.585047ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
31492024-08-05T16:46:11.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 1.27063516s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
31502024-08-05T16:46:11.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 814.596329ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
31512024-08-05T16:46:11.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.132887289s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
31522024-08-05T16:46:11.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.132179173s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
31532024-08-05T16:46:11.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 647.455545ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
31542024-08-05T16:46:12.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 574.019624ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
31552024-08-05T16:46:12.199ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 841.536718ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
31562024-08-05T16:46:12.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 777.012962ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
31572024-08-05T16:46:12.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.366685835s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
31582024-08-05T16:46:12.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 802.588514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
31592024-08-05T16:46:12.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 1.372439318s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
31602024-08-05T16:46:12.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 823.557388ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
31612024-08-05T16:46:12.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 596.575708ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
31622024-08-05T16:46:12.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 578.79616ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
31632024-08-05T16:46:12.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 638.059813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
31642024-08-05T16:46:12.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.213590063s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
31652024-08-05T16:46:12.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 745.049475ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
31662024-08-05T16:46:12.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 1.441450282s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
31672024-08-05T16:46:12.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.335298657s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
31682024-08-05T16:46:13.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.33788936s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
31692024-08-05T16:46:13.011ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31702024-08-05T16:46:13.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 767.767095ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
31712024-08-05T16:46:13.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 1.373752933s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
31722024-08-05T16:46:13.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 1.209784242s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
31732024-08-05T16:46:13.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 574.621239ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
31742024-08-05T16:46:13.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 1.189748832s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
31752024-08-05T16:46:13.149ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65 zone file = illumos-utils/src/running_zone.rs:472 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
31762024-08-05T16:46:13.171ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 701.989758ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
31772024-08-05T16:46:13.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 801.618713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
31782024-08-05T16:46:13.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.46047574s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
31792024-08-05T16:46:13.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 44.111472ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
31802024-08-05T16:46:13.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.281072482s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
31812024-08-05T16:46:13.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 100.418621ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
31822024-08-05T16:46:13.465ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 664.705953ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
31832024-08-05T16:46:13.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 1.493094619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
31842024-08-05T16:46:13.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 203.917179ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
31852024-08-05T16:46:13.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.233753766s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
31862024-08-05T16:46:13.720ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 663.332208ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
31872024-08-05T16:46:13.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 325.821283ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
31882024-08-05T16:46:13.835ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.291128782s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
31892024-08-05T16:46:13.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 986.89443ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
31902024-08-05T16:46:13.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 883.254041ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
31912024-08-05T16:46:14.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 764.683115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
31922024-08-05T16:46:14.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.253050717s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
31932024-08-05T16:46:14.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 459.513442ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
31942024-08-05T16:46:14.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 1.375196395s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
31952024-08-05T16:46:14.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 934.886293ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
31962024-08-05T16:46:14.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.193627024s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
31972024-08-05T16:46:14.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 786.387762ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
31982024-08-05T16:46:14.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 658.084726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
31992024-08-05T16:46:14.364ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 969.789439ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
32002024-08-05T16:46:14.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 782.459381ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
32012024-08-05T16:46:14.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 817.845242ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
32022024-08-05T16:46:14.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 1.12988078s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
32032024-08-05T16:46:14.582ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 522.727209ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
32042024-08-05T16:46:14.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.103538036s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
32052024-08-05T16:46:14.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 524.256118ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
32062024-08-05T16:46:14.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 907.752662ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
32072024-08-05T16:46:14.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 1.378934544s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
32082024-08-05T16:46:14.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 545.356006ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
32092024-08-05T16:46:14.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 888.377632ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
32102024-08-05T16:46:15.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 724.433731ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
32112024-08-05T16:46:15.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 516.68633ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
32122024-08-05T16:46:15.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.13480986s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
32132024-08-05T16:46:15.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 1.188620425s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
32142024-08-05T16:46:15.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 848.958873ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
32152024-08-05T16:46:15.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 1.185622753s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
32162024-08-05T16:46:15.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 1.058158921s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
32172024-08-05T16:46:15.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 581.947302ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
32182024-08-05T16:46:15.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.414529511s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
32192024-08-05T16:46:15.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 628.548151ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
32202024-08-05T16:46:15.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.382095136s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
32212024-08-05T16:46:15.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.396459598s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
32222024-08-05T16:46:15.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 1.225663739s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
32232024-08-05T16:46:15.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 894.722642ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
32242024-08-05T16:46:15.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 1.341417838s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
32252024-08-05T16:46:15.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 1.413315429s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
32262024-08-05T16:46:15.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 807.002476ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
32272024-08-05T16:46:15.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 636.96929ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
32282024-08-05T16:46:15.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 634.958691ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
32292024-08-05T16:46:15.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 514.190236ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
32302024-08-05T16:46:15.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 963.353281ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
32312024-08-05T16:46:15.956ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.449713179s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
32322024-08-05T16:46:15.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.493563491s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
32332024-08-05T16:46:16.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 851.927892ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
32342024-08-05T16:46:16.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.487281148s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
32352024-08-05T16:46:16.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 699.052716ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
32362024-08-05T16:46:16.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 1.240356763s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
32372024-08-05T16:46:16.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 1.214463379s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
32382024-08-05T16:46:16.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 540.800493ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
32392024-08-05T16:46:16.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 1.084986978s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
32402024-08-05T16:46:16.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 1.384564782s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
32412024-08-05T16:46:16.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.195954844s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
32422024-08-05T16:46:16.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.428631706s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
32432024-08-05T16:46:16.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 717.93903ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
32442024-08-05T16:46:16.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.429260039s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
32452024-08-05T16:46:16.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 957.898844ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
32462024-08-05T16:46:16.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.457887502s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
32472024-08-05T16:46:16.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.050942502s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
32482024-08-05T16:46:16.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 932.47802ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
32492024-08-05T16:46:16.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 531.428139ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
32502024-08-05T16:46:16.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 888.439994ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
32512024-08-05T16:46:17.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 904.5932ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
32522024-08-05T16:46:17.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 749.058483ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
32532024-08-05T16:46:17.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.274763288s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
32542024-08-05T16:46:17.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 856.102219ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
32552024-08-05T16:46:17.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 1.402900706s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
32562024-08-05T16:46:17.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 1.182215481s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
32572024-08-05T16:46:17.497ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 639.764301ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
32582024-08-05T16:46:17.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 826.014779ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
32592024-08-05T16:46:17.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 865.978194ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
32602024-08-05T16:46:17.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 741.200994ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
32612024-08-05T16:46:17.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 717.107003ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
32622024-08-05T16:46:17.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 914.886296ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
32632024-08-05T16:46:17.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 745.319253ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
32642024-08-05T16:46:17.830ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 769.722777ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
32652024-08-05T16:46:17.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 1.364703415s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
32662024-08-05T16:46:17.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 988.934834ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
32672024-08-05T16:46:17.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.36672005s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
32682024-08-05T16:46:17.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 1.185431347s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
32692024-08-05T16:46:17.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.423363704s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
32702024-08-05T16:46:18.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 763.46773ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
32712024-08-05T16:46:18.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.417375208s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
32722024-08-05T16:46:18.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 666.469597ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
32732024-08-05T16:46:18.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.061981185s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
32742024-08-05T16:46:18.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 781.450853ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
32752024-08-05T16:46:18.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 1.331357579s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
32762024-08-05T16:46:18.452ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 692.156821ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
32772024-08-05T16:46:18.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 1.388676427s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
32782024-08-05T16:46:18.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 633.134121ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
32792024-08-05T16:46:18.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 1.063747287s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
32802024-08-05T16:46:18.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.260851761s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
32812024-08-05T16:46:18.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 1.462154192s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
32822024-08-05T16:46:18.702ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.063535673s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
32832024-08-05T16:46:18.857ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 1.052479997s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
32842024-08-05T16:46:18.878ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 995.445115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
32852024-08-05T16:46:18.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.022656339s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
32862024-08-05T16:46:18.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 718.504359ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
32872024-08-05T16:46:19.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 793.86676ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
32882024-08-05T16:46:19.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 566.379828ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
32892024-08-05T16:46:19.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.13998943s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
32902024-08-05T16:46:19.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 1.275136395s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
32912024-08-05T16:46:19.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 1.091742577s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
32922024-08-05T16:46:19.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.322055677s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
32932024-08-05T16:46:19.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 812.822675ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
32942024-08-05T16:46:19.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.149948708s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
32952024-08-05T16:46:19.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 735.582906ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
32962024-08-05T16:46:19.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.00308478s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
32972024-08-05T16:46:19.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 1.0639422s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
32982024-08-05T16:46:19.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 1.483897808s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
32992024-08-05T16:46:19.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 1.42237719s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
33002024-08-05T16:46:19.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.26778368s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
33012024-08-05T16:46:19.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 563.800479ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
33022024-08-05T16:46:19.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 695.100052ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
33032024-08-05T16:46:19.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 612.565246ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
33042024-08-05T16:46:19.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.338915523s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
33052024-08-05T16:46:19.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 679.878591ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
33062024-08-05T16:46:19.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 717.721195ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
33072024-08-05T16:46:20.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 828.888732ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
33082024-08-05T16:46:20.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 559.794319ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
33092024-08-05T16:46:20.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 885.785219ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
33102024-08-05T16:46:20.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 650.015616ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
33112024-08-05T16:46:20.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.493059892s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
33122024-08-05T16:46:20.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 790.800348ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
33132024-08-05T16:46:20.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 1.08442228s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
33142024-08-05T16:46:20.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 664.741362ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
33152024-08-05T16:46:20.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 1.378117357s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
33162024-08-05T16:46:20.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 658.969822ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
33172024-08-05T16:46:20.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d") failed: Property not found. retry in 1.451688052s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
33182024-08-05T16:46:20.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 520.969468ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
33192024-08-05T16:46:20.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 591.116015ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
33202024-08-05T16:46:20.703ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 1.19193164s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
33212024-08-05T16:46:20.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 881.717369ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
33222024-08-05T16:46:20.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.414747353s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
33232024-08-05T16:46:20.996ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 541.590309ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
33242024-08-05T16:46:21.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 708.672423ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
33252024-08-05T16:46:21.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.11896247s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
33262024-08-05T16:46:21.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.406907141s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
33272024-08-05T16:46:21.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 1.183675482s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
33282024-08-05T16:46:21.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 1.39317182s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
33292024-08-05T16:46:21.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 710.580722ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
33302024-08-05T16:46:21.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 1.314509011s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
33312024-08-05T16:46:21.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 768.492737ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
33322024-08-05T16:46:21.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 921.072844ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
33332024-08-05T16:46:21.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 965.578863ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
33342024-08-05T16:46:21.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.246690013s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
33352024-08-05T16:46:21.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da") failed: Property not found. retry in 1.078685702s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
33362024-08-05T16:46:21.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb") failed: Property not found. retry in 923.111793ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
33372024-08-05T16:46:21.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 755.808918ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
33382024-08-05T16:46:21.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 1.15758311s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
33392024-08-05T16:46:21.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015") failed: Property not found. retry in 1.321754029s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
33402024-08-05T16:46:21.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 1.352798731s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
33412024-08-05T16:46:21.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 719.074246ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
33422024-08-05T16:46:21.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 1.491803865s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
33432024-08-05T16:46:22.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7") failed: Property not found. retry in 1.345168387s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
33442024-08-05T16:46:22.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 745.514708ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
33452024-08-05T16:46:22.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 947.372926ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
33462024-08-05T16:46:22.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.363076282s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
33472024-08-05T16:46:22.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.175814193s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
33482024-08-05T16:46:22.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 814.403688ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
33492024-08-05T16:46:22.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 915.480197ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
33502024-08-05T16:46:22.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.103925563s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
33512024-08-05T16:46:22.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1") failed: Property not found. retry in 753.489888ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
33522024-08-05T16:46:22.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.258726544s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
33532024-08-05T16:46:22.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 655.698972ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
33542024-08-05T16:46:22.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 767.259643ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
33552024-08-05T16:46:22.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300") failed: Property not found. retry in 1.379343321s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
33562024-08-05T16:46:22.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.098760675s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
33572024-08-05T16:46:23.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 820.986561ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
33582024-08-05T16:46:23.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a") failed: Property not found. retry in 1.248638652s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
33592024-08-05T16:46:23.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 550.665064ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
33602024-08-05T16:46:23.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 801.726727ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
33612024-08-05T16:46:23.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 597.110841ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
33622024-08-05T16:46:23.558ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069") failed: Property not found. retry in 835.907411ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
33632024-08-05T16:46:23.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451") failed: Property not found. retry in 1.199775322s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
33642024-08-05T16:46:23.585ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 774.187403ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
33652024-08-05T16:46:23.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf") failed: Property not found. retry in 1.06990322s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
33662024-08-05T16:46:23.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 1.069432251s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
33672024-08-05T16:46:23.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8") failed: Property not found. retry in 1.262450996s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
33682024-08-05T16:46:24.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8") failed: Property not found. retry in 1.384170881s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
33692024-08-05T16:46:24.095ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631") failed: Property not found. retry in 1.480893566s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
33702024-08-05T16:46:24.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 550.787288ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
33712024-08-05T16:46:24.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65") failed: Property not found. retry in 750.396955ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
33722024-08-05T16:46:24.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83") failed: Property not found. retry in 960.565465ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
33732024-08-05T16:46:24.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42") failed: Property not found. retry in 587.720129ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
33742024-08-05T16:46:24.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d") failed: Property not found. retry in 829.968088ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
33752024-08-05T16:46:24.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2") failed: Property not found. retry in 663.079445ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
33762024-08-05T16:46:25.345ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 62.221185713s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
33772024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_nexus_1c82ddd6-6f5a-4a92-a9b3-fde01c046c42
33782024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_oximeter_893afdd2-39c6-4f94-b381-facb1a40bbb1
33792024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_pantry_c02f008f-19c7-48c5-93c6-3f776c51e015
33802024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_external_dns_8995285b-5cb0-4d3a-b021-8aec05314a65
33812024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_external_dns_8a344ff3-4023-4f5e-a29a-3db2dc6980b8
33822024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_clickhouse_061f99f8-6626-4a81-8262-abd77ceeb36a
33832024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_pantry_738351b7-dacb-486a-ab38-cf8aefaef9cb
33842024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_29db7ccb-90e1-414b-aeee-a9393f639631
33852024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_8029adce-2a1d-48e9-a435-7bcff4f3bb83
33862024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_48f67680-6ea6-4e0c-9968-c45baf393c1d
33872024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_58b140e5-f61e-4410-8851-af929dfd8069
33882024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_f5fbccc2-b8b1-4dd2-9655-42d226ad2451
33892024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_19f0f364-8840-4b44-9eea-e3a6f21b6300
33902024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_c1741444-7ab7-4888-aa3c-e7b6da3527da
33912024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_nexus_866d6328-4ca1-435e-8a98-cd5268df29e8
33922024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_d857801a-a95b-4937-a2f0-31ad7c239ac7
33932024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_0d925b8a-ac40-4a91-9334-d6f7d7a29fb2
33942024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_nexus_4952ec5d-23ba-48ea-9509-011fcb02adaf
33952024-08-05T16:46:25.615ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3061 zone = oxz_crucible_pantry_15711896-d3e4-44e3-b330-624f0f5e064d
33962024-08-05T16:46:25.617ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33972024-08-05T16:46:25.642ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33982024-08-05T16:46:25.657ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 36653109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59148 req_id = 5a24faad-e026-416a-bfc3-acaab4c101c5 response_code = 204 uri = /omicron-zones
33992024-08-05T16:46:25.692ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1250
34002024-08-05T16:46:25.692ZINFOSledAgent (RSS): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
34012024-08-05T16:46:25.692ZINFOSledAgent (RSS): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
34022024-08-05T16:46:25.705ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:687
34032024-08-05T16:46:25.712ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:700
34042024-08-05T16:46:25.786ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:831
34052024-08-05T16:46:25.826ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34062024-08-05T16:46:25.914ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34072024-08-05T16:46:26.074ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34082024-08-05T16:46:26.257ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34092024-08-05T16:46:26.434ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34102024-08-05T16:46:26.697ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34112024-08-05T16:46:26.891ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34122024-08-05T16:46:27.049ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34132024-08-05T16:46:27.328ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34142024-08-05T16:46:27.794ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34152024-08-05T16:46:28.547ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34162024-08-05T16:46:29.275ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34172024-08-05T16:46:29.878ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34182024-08-05T16:46:30.940ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34192024-08-05T16:46:30.963ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56547
34202024-08-05T16:46:30.965ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 739 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56547 req_id = f0218010-ae46-4d07-bc51-2e3a6250e601 response_code = 200 uri = /baseboard
34212024-08-05T16:46:31.084ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34222024-08-05T16:46:31.660ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34232024-08-05T16:46:31.951ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34242024-08-05T16:46:32.485ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34252024-08-05T16:46:32.958ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34262024-08-05T16:46:33.244ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34272024-08-05T16:46:33.960ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34282024-08-05T16:46:33.986ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34292024-08-05T16:46:34.969ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34302024-08-05T16:46:35.239ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34312024-08-05T16:46:35.972ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34322024-08-05T16:46:36.106ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34332024-08-05T16:46:36.978ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34342024-08-05T16:46:37.239ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34352024-08-05T16:46:37.669ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34362024-08-05T16:46:37.981ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34372024-08-05T16:46:38.984ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34382024-08-05T16:46:39.240ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34392024-08-05T16:46:39.906ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34402024-08-05T16:46:39.987ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34412024-08-05T16:46:40.990ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34422024-08-05T16:46:41.239ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34432024-08-05T16:46:41.993ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34442024-08-05T16:46:42.995ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:225 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34452024-08-05T16:46:43.240ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/3abf3a4f-8820-40af-a01f-4369b42a5529): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:601 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34462024-08-05T16:46:43.417ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/b682b2a4-3c99-42b2-9e14-9bba53281f21/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:893
34472024-08-05T16:46:46.509ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "14e941f4-8867-4455-8b04-08b8c23fa66f", "content-length": "173", "date": "Mon, 05 Aug 2024 16:46:46 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"3abf3a4f-8820-40af-a01f-4369b42a5529\"", request_id: "14e941f4-8867-4455-8b04-08b8c23fa66f" } file = sled-agent/src/rack_setup/service.rs:893
34482024-08-05T16:46:47.274ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:574 new_generation = 1 old_generation = 1 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
34492024-08-05T16:46:47.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62785
34502024-08-05T16:46:47.412ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
34512024-08-05T16:46:47.504ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 93735 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62785 req_id = 2fbfae6b-15b2-47e1-8755-39b0e36698bd response_code = 200 uri = /inventory
34522024-08-05T16:46:47.515ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
34532024-08-05T16:46:47.515ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
34542024-08-05T16:46:47.516ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
34552024-08-05T16:46:47.517ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 12338 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62785 req_id = cd5802e3-b05b-4ac4-9a3c-b8545e124e9d response_code = 200 uri = /omicron-zones
34562024-08-05T16:46:49.547ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54671
34572024-08-05T16:46:49.551ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34582024-08-05T16:46:49.552ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34592024-08-05T16:46:49.553ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34602024-08-05T16:46:49.553ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34612024-08-05T16:46:49.553ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
34622024-08-05T16:46:49.553ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34632024-08-05T16:46:49.553ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34642024-08-05T16:46:49.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 5014 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54671 req_id = 0ef38b26-3e64-44ed-a802-90fa1ad1fb4b response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34652024-08-05T16:46:49.554ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:903
34662024-08-05T16:46:49.852ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33060
34672024-08-05T16:46:49.852ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
34682024-08-05T16:46:49.940ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 87041 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33060 req_id = 3ef1809a-a80f-422a-a500-9a386d396217 response_code = 200 uri = /inventory
34692024-08-05T16:46:49.940ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57546
34702024-08-05T16:46:49.940ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
34712024-08-05T16:46:49.940ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/config/omicron-zones.json file = common/src/ledger.rs:177
34722024-08-05T16:46:49.940ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34732024-08-05T16:46:49.940ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/config/omicron-zones.json file = common/src/ledger.rs:177
34742024-08-05T16:46:49.940ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34752024-08-05T16:46:49.940ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34762024-08-05T16:46:49.940ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34772024-08-05T16:46:49.940ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
34782024-08-05T16:46:49.940ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34792024-08-05T16:46:49.940ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34802024-08-05T16:46:49.940ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 578 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57546 req_id = 6f642107-d751-49f0-a370-70a0863c6ccb response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34812024-08-05T16:46:49.941ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 864 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33060 req_id = fac316b5-7c4e-489b-acb2-8c5bbbb66a11 response_code = 200 uri = /omicron-zones
34822024-08-05T16:46:49.963ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47565
34832024-08-05T16:46:49.964ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34842024-08-05T16:46:49.964ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34852024-08-05T16:46:49.964ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34862024-08-05T16:46:49.964ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34872024-08-05T16:46:49.964ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
34882024-08-05T16:46:49.964ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34892024-08-05T16:46:49.964ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34902024-08-05T16:46:49.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 390 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47565 req_id = 59b23a5f-3478-4887-bfd1-a5b6001025a3 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34912024-08-05T16:46:49.987ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37350
34922024-08-05T16:46:49.988ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34932024-08-05T16:46:49.988ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34942024-08-05T16:46:49.988ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34952024-08-05T16:46:49.988ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34962024-08-05T16:46:49.988ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
34972024-08-05T16:46:49.988ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34982024-08-05T16:46:49.988ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34992024-08-05T16:46:49.988ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 294 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:37350 req_id = 0a65042b-735e-425f-be82-56ba8ec35f45 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
35002024-08-05T16:46:59.455ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50984
35012024-08-05T16:46:59.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 786 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50984 req_id = 90715199-5761-4bc9-80e3-3a4a257ba075 response_code = 200 uri = /vpc-routes
35022024-08-05T16:46:59.519ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 4010 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50984 req_id = 77ec3fce-079f-4919-902d-7ddc9f4584c8 response_code = 204 uri = /vpc-routes
35032024-08-05T16:47:00.997ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51624
35042024-08-05T16:47:00.997ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:51624 req_id = 76942a5b-db7e-435f-87fd-5191845a7b5e response_code = 200 uri = /baseboard
35052024-08-05T16:47:15.125ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38501
35062024-08-05T16:47:15.125ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38501 req_id = 7587186e-8bfd-4fe2-97b5-345f89fb3f56 response_code = 200 uri = /vpc-routes
35072024-08-05T16:47:15.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:38501 req_id = ae082a3d-e6f2-41e7-b6f2-a3d8f3aa30c1 response_code = 204 uri = /vpc-routes
35082024-08-05T16:47:15.232ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46236
35092024-08-05T16:47:15.232ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46236 req_id = 6a872ec8-d41f-419e-9031-f599774f3c1a response_code = 200 uri = /vpc-routes
35102024-08-05T16:47:15.241ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:46236 req_id = 3c61418e-33e4-45a2-ae9a-aaee7d5f563b response_code = 204 uri = /vpc-routes
35112024-08-05T16:47:15.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51151
35122024-08-05T16:47:15.261ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 825 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51151 req_id = fe46ba4e-9321-488b-bee8-40a9c30b5370 response_code = 200 uri = /vpc-routes
35132024-08-05T16:47:15.269ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51151 req_id = 53f92e94-bdca-4c4b-9ded-0a0a27f817d1 response_code = 204 uri = /vpc-routes
35142024-08-05T16:47:15.269ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50348
35152024-08-05T16:47:15.270ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 398 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50348 req_id = 3c03ea6b-46dc-449d-9297-36f687ca0847 response_code = 200 uri = /v2p
35162024-08-05T16:47:15.286ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41855
35172024-08-05T16:47:15.287ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41855 req_id = f0dfbd1f-4565-4b76-8e40-2ed296fffd69 response_code = 200 uri = /v2p
35182024-08-05T16:47:15.406ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36320
35192024-08-05T16:47:15.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 652 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36320 req_id = 1056bf2e-54e1-44ac-be91-eac10067dc5c response_code = 200 uri = /v2p
35202024-08-05T16:47:15.440ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36293
35212024-08-05T16:47:15.443ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
35222024-08-05T16:47:15.505ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
35232024-08-05T16:47:15.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 91651 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36293 req_id = 707fe93c-d5ba-4280-aba4-47688d037222 response_code = 204 uri = /switch-ports
35242024-08-05T16:47:15.532ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57167
35252024-08-05T16:47:15.532ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48314
35262024-08-05T16:47:15.533ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
35272024-08-05T16:47:15.539ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 6205 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36293 req_id = d861fabe-0d1c-4ddd-8045-a10a9a201fb1 response_code = 200 uri = /network-bootstore-config
35282024-08-05T16:47:15.559ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
35292024-08-05T16:47:15.565ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
35302024-08-05T16:47:15.566ZINFOSledAgent (bootstore): Writing ledger to /pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
35312024-08-05T16:47:15.567ZINFOSledAgent (bootstore): Writing ledger to /pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
35322024-08-05T16:47:15.568ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
35332024-08-05T16:47:15.568ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3386 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36293 req_id = e436fdf4-c88e-465d-9a1c-b2c68308bd22 response_code = 204 uri = /network-bootstore-config
35342024-08-05T16:47:15.582ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
35352024-08-05T16:47:15.582ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 49813 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:57167 req_id = e42ad4fe-b412-4b84-9d9a-608aa528f908 response_code = 204 uri = /switch-ports
35362024-08-05T16:47:15.586ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 624 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57167 req_id = a016a64b-4565-4c5d-b5cd-d72aeeb1fd8f response_code = 200 uri = /network-bootstore-config
35372024-08-05T16:47:15.608ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
35382024-08-05T16:47:15.632ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 99651 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48314 req_id = f2bdf0d1-edcb-46f1-9468-3884dbcd6f4c response_code = 204 uri = /switch-ports
35392024-08-05T16:47:15.633ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48314 req_id = 92f1415c-9067-45ca-9fea-f1e25b7c338e response_code = 200 uri = /network-bootstore-config
35402024-08-05T16:47:31.030ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42922
35412024-08-05T16:47:31.030ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:42922 req_id = 6273ba72-1864-40f0-ad87-f4416e51b464 response_code = 200 uri = /baseboard
35422024-08-05T16:47:45.264ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33138
35432024-08-05T16:47:45.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 197 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33138 req_id = 7c2c44e7-7248-4006-bf43-66488c0b2355 response_code = 200 uri = /v2p
35442024-08-05T16:47:45.290ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33120
35452024-08-05T16:47:45.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33120 req_id = 8bec70a4-3506-4a60-8247-8c32872e6a18 response_code = 200 uri = /vpc-routes
35462024-08-05T16:47:45.299ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33120 req_id = e877faa7-485e-444e-89a2-913069364bd6 response_code = 204 uri = /vpc-routes
35472024-08-05T16:47:45.331ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64806
35482024-08-05T16:47:45.332ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 581 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64806 req_id = f26fad78-d6ec-4ab4-89d7-94543b4fe668 response_code = 200 uri = /vpc-routes
35492024-08-05T16:47:45.341ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1051 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64806 req_id = d353de38-760a-4c6c-84e1-ecd64ebfa5a5 response_code = 204 uri = /vpc-routes
35502024-08-05T16:47:45.442ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60844
35512024-08-05T16:47:45.442ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60844 req_id = cf58d1ea-ecf8-437c-880c-583cb0ef9acc response_code = 200 uri = /vpc-routes
35522024-08-05T16:47:45.450ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60844 req_id = 6cacbb3c-733a-475d-8752-fe3f1a885156 response_code = 204 uri = /vpc-routes
35532024-08-05T16:47:45.481ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40744
35542024-08-05T16:47:45.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1082 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40744 req_id = bf4a66a6-33ed-4157-9edf-3d958b605506 response_code = 200 uri = /v2p
35552024-08-05T16:47:45.506ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44969
35562024-08-05T16:47:45.506ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44969 req_id = 61d8640d-47b0-45b7-8c80-a957415cb111 response_code = 200 uri = /v2p
35572024-08-05T16:47:45.616ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44794
35582024-08-05T16:47:45.616ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
35592024-08-05T16:47:45.642ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
35602024-08-05T16:47:45.663ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 47544 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44794 req_id = 252b181f-d1b0-4778-ad94-ec53ec487bf8 response_code = 204 uri = /switch-ports
35612024-08-05T16:47:45.664ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53546
35622024-08-05T16:47:45.664ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38182
35632024-08-05T16:47:45.664ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
35642024-08-05T16:47:45.667ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3599 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44794 req_id = 423694f3-bf43-49e4-88ac-b9ea5acbd228 response_code = 200 uri = /network-bootstore-config
35652024-08-05T16:47:45.689ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
35662024-08-05T16:47:45.711ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
35672024-08-05T16:47:45.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 47810 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53546 req_id = 948ce42d-48f6-4e24-9759-b3ce5ebd6259 response_code = 204 uri = /switch-ports
35682024-08-05T16:47:45.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 649 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53546 req_id = 394f62a1-12ba-477b-bb16-b2a3dc05e92a response_code = 200 uri = /network-bootstore-config
35692024-08-05T16:47:45.738ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
35702024-08-05T16:47:45.759ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 95673 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38182 req_id = 9d8c7e5c-35e6-405a-9919-309c484340ba response_code = 204 uri = /switch-ports
35712024-08-05T16:47:45.760ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 182 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38182 req_id = dc8ed8e8-9d1f-4af8-b69d-55cf03449bd4 response_code = 200 uri = /network-bootstore-config
35722024-08-05T16:47:47.246ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
35732024-08-05T16:47:57.701ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:40509 remote_addr = [fd00:1122:3344:101::d]:56611 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529
35742024-08-05T16:47:57.710ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 8629 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = 76dbeeca-6037-4445-8b18-86638edb1a9a response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
35752024-08-05T16:48:01.063ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39187
35762024-08-05T16:48:01.064ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 589 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39187 req_id = cfbc08a1-c645-4821-b2c0-51132197bc5e response_code = 200 uri = /baseboard
35772024-08-05T16:48:15.091ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50925
35782024-08-05T16:48:15.092ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1150 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50925 req_id = 51327236-6626-4c33-b1c3-040dbb48f874 response_code = 200 uri = /vpc-routes
35792024-08-05T16:48:15.100ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50925 req_id = c1717e0f-e37c-42fa-9437-5a07bba7488c response_code = 204 uri = /vpc-routes
35802024-08-05T16:48:15.145ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64742
35812024-08-05T16:48:15.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64742 req_id = f449888c-922f-4173-96b8-d477492514a4 response_code = 200 uri = /vpc-routes
35822024-08-05T16:48:15.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64742 req_id = f5183f4a-015b-4409-993a-1962dafd376d response_code = 204 uri = /vpc-routes
35832024-08-05T16:48:15.201ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43812
35842024-08-05T16:48:15.202ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 789 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43812 req_id = 519295af-5b23-4848-baf0-4d26062f5547 response_code = 200 uri = /vpc-routes
35852024-08-05T16:48:15.210ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43812 req_id = b3607089-3f25-4009-ad69-3f9ae428e0cf response_code = 204 uri = /vpc-routes
35862024-08-05T16:48:15.214ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54818
35872024-08-05T16:48:15.215ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54818 req_id = 2fa92a68-5f45-4f02-95bc-24a8004ddece response_code = 200 uri = /v2p
35882024-08-05T16:48:15.234ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41021
35892024-08-05T16:48:15.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41021 req_id = 3635fdc5-64e5-4a50-98cb-389611d1cc69 response_code = 200 uri = /v2p
35902024-08-05T16:48:15.279ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43991
35912024-08-05T16:48:15.279ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
35922024-08-05T16:48:15.305ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
35932024-08-05T16:48:15.326ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46803 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43991 req_id = 335a7ab2-af6e-48a2-b38f-61ff174968e0 response_code = 204 uri = /switch-ports
35942024-08-05T16:48:15.326ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45915
35952024-08-05T16:48:15.327ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 204 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45915 req_id = f891c9a5-b4cf-4599-9158-0c61a4537239 response_code = 200 uri = /v2p
35962024-08-05T16:48:15.327ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 887 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43991 req_id = 4eca3c1b-71dd-4dfe-b8ce-d89ccccf53e8 response_code = 200 uri = /network-bootstore-config
35972024-08-05T16:48:15.329ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40400
35982024-08-05T16:48:15.330ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
35992024-08-05T16:48:15.351ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36002024-08-05T16:48:15.371ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 41793 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40400 req_id = c5e702f8-17c6-47ab-be81-fb323af6e4e2 response_code = 204 uri = /switch-ports
36012024-08-05T16:48:15.371ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54293
36022024-08-05T16:48:15.372ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36032024-08-05T16:48:15.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 2811 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40400 req_id = feb32142-fdf6-4eda-817f-bb565b5826e9 response_code = 200 uri = /network-bootstore-config
36042024-08-05T16:48:15.393ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36052024-08-05T16:48:15.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 42775 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54293 req_id = 5ab536f8-9511-49e4-9b6d-77260a73029b response_code = 204 uri = /switch-ports
36062024-08-05T16:48:15.415ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54293 req_id = 417441c1-aab0-49af-a26c-3f606ad6a59f response_code = 200 uri = /network-bootstore-config
36072024-08-05T16:48:27.701ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1186 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = 70b35226-a9f9-4835-a916-a251f3f388f1 response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
36082024-08-05T16:48:31.097ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62362
36092024-08-05T16:48:31.097ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 64 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62362 req_id = a88d1c07-c2fe-460f-9039-26f273920bd0 response_code = 200 uri = /baseboard
36102024-08-05T16:48:45.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54451
36112024-08-05T16:48:45.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54451 req_id = f56c5ae7-2366-4d22-a4a1-872325ea38de response_code = 200 uri = /vpc-routes
36122024-08-05T16:48:45.198ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54451 req_id = fd27d3b2-35f2-4945-ade9-5eff1bfc207c response_code = 204 uri = /vpc-routes
36132024-08-05T16:48:45.253ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58957
36142024-08-05T16:48:45.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58957 req_id = 8b0e34de-cc78-4379-bfa8-cb51732ce565 response_code = 200 uri = /vpc-routes
36152024-08-05T16:48:45.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53933
36162024-08-05T16:48:45.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53933 req_id = 491e618d-121f-4044-84d8-406e0f0828a4 response_code = 200 uri = /vpc-routes
36172024-08-05T16:48:45.261ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41091
36182024-08-05T16:48:45.263ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1593 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41091 req_id = d67d8530-058c-417b-897f-acff5214559d response_code = 200 uri = /v2p
36192024-08-05T16:48:45.263ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1467 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58957 req_id = b8b84f23-8574-445a-9e6b-5bf67a572aec response_code = 204 uri = /vpc-routes
36202024-08-05T16:48:45.268ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53933 req_id = f5c1fbc9-d402-46e0-ade2-08a04bc47737 response_code = 204 uri = /vpc-routes
36212024-08-05T16:48:45.276ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63956
36222024-08-05T16:48:45.277ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63956 req_id = b5536c2a-19bc-4c2e-a659-343ad13d8539 response_code = 200 uri = /v2p
36232024-08-05T16:48:45.341ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60885
36242024-08-05T16:48:45.341ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60885 req_id = 1af94936-b2cb-48bb-9575-c071ad0e1466 response_code = 200 uri = /v2p
36252024-08-05T16:48:45.511ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34560
36262024-08-05T16:48:45.511ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36272024-08-05T16:48:45.536ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36282024-08-05T16:48:45.560ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 48805 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34560 req_id = 86ee9f10-d2e7-495e-8d68-0734cd1a8c80 response_code = 204 uri = /switch-ports
36292024-08-05T16:48:45.560ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60507
36302024-08-05T16:48:45.560ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36312024-08-05T16:48:45.563ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3289 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34560 req_id = 221ff496-afab-4d5e-a26f-a97502a8c14f response_code = 200 uri = /network-bootstore-config
36322024-08-05T16:48:45.583ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32844
36332024-08-05T16:48:45.585ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36342024-08-05T16:48:45.606ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36352024-08-05T16:48:45.606ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 45807 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60507 req_id = 24a21b38-cf83-4454-9ad7-78f0e2434856 response_code = 204 uri = /switch-ports
36362024-08-05T16:48:45.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 405 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60507 req_id = 531bbb60-71be-43be-b1ed-59d37b465622 response_code = 200 uri = /network-bootstore-config
36372024-08-05T16:48:45.627ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36382024-08-05T16:48:45.648ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 64707 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:32844 req_id = 3518ad88-809c-4ad7-a7e3-a0dc1b2ae66c response_code = 204 uri = /switch-ports
36392024-08-05T16:48:45.648ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32844 req_id = df5171a2-20cf-4050-8e3c-c0a8d3c9a800 response_code = 200 uri = /network-bootstore-config
36402024-08-05T16:48:57.701ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 837 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = 7a2d76c3-e234-4f00-8142-85314768d3c8 response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
36412024-08-05T16:49:01.129ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41664
36422024-08-05T16:49:01.129ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 34 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41664 req_id = 2d5a930d-b8ed-4d17-9da8-34295d6728aa response_code = 200 uri = /baseboard
36432024-08-05T16:49:15.076ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51198
36442024-08-05T16:49:15.076ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51198 req_id = eed75bf8-f747-4adf-ac1c-36d3e957bbe1 response_code = 200 uri = /vpc-routes
36452024-08-05T16:49:15.085ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1187 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51198 req_id = 206528f2-eec2-4c35-8bf4-b7d56b218853 response_code = 204 uri = /vpc-routes
36462024-08-05T16:49:15.133ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61754
36472024-08-05T16:49:15.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 26 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61754 req_id = f37ff7d7-6d91-4467-85b2-fbb745561b75 response_code = 200 uri = /vpc-routes
36482024-08-05T16:49:15.141ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1341 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61754 req_id = 597efe00-be9c-4e7f-8ffe-f3197b4484eb response_code = 204 uri = /vpc-routes
36492024-08-05T16:49:15.198ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53168
36502024-08-05T16:49:15.198ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53168 req_id = 5a6b52b5-0125-49f5-8578-32ba3ecd909d response_code = 200 uri = /vpc-routes
36512024-08-05T16:49:15.205ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1147 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53168 req_id = 5e00213a-d1d8-4e68-a9a4-c715b7306c65 response_code = 204 uri = /vpc-routes
36522024-08-05T16:49:15.207ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36407
36532024-08-05T16:49:15.207ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36407 req_id = de500518-8efb-4cbc-8364-2ba6fff00651 response_code = 200 uri = /v2p
36542024-08-05T16:49:15.235ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40685
36552024-08-05T16:49:15.235ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40685 req_id = c561a37b-4a10-4ab7-ba50-dab6fbf70e3f response_code = 200 uri = /v2p
36562024-08-05T16:49:15.309ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36240
36572024-08-05T16:49:15.310ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36582024-08-05T16:49:15.334ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36592024-08-05T16:49:15.358ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 48101 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36240 req_id = 5f687014-3395-40dc-9f6c-7eb732b60d41 response_code = 204 uri = /switch-ports
36602024-08-05T16:49:15.358ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59690
36612024-08-05T16:49:15.358ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59690 req_id = cd404043-a3ad-4ed4-8fad-e6db06b0dcc1 response_code = 200 uri = /v2p
36622024-08-05T16:49:15.358ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 208 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36240 req_id = 3c632655-22f1-4425-9b5f-51a0d86f362a response_code = 200 uri = /network-bootstore-config
36632024-08-05T16:49:15.385ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46054
36642024-08-05T16:49:15.386ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36652024-08-05T16:49:15.409ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36662024-08-05T16:49:15.431ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 45040 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46054 req_id = 92f47cb4-dd5b-4333-9bb3-51cf9d0c64d7 response_code = 204 uri = /switch-ports
36672024-08-05T16:49:15.431ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46054 req_id = f99d436e-5596-4be0-a64a-8f2df0b5efbe response_code = 200 uri = /network-bootstore-config
36682024-08-05T16:49:15.432ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53165
36692024-08-05T16:49:15.432ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36702024-08-05T16:49:15.456ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36712024-08-05T16:49:15.477ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 45261 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53165 req_id = e3c73e91-8eea-4492-b8c0-4457fa18c043 response_code = 204 uri = /switch-ports
36722024-08-05T16:49:15.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 572 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53165 req_id = 516ce4dd-039a-4311-ac4c-218391d78752 response_code = 200 uri = /network-bootstore-config
36732024-08-05T16:49:27.701ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 647 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = 2df2b0db-3fdb-4b9b-a958-331c9ac29015 response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
36742024-08-05T16:49:31.161ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53390
36752024-08-05T16:49:31.161ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:53390 req_id = f858f876-6346-452e-96e6-404914cd9eb9 response_code = 200 uri = /baseboard
36762024-08-05T16:49:45.206ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42427
36772024-08-05T16:49:45.207ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42427 req_id = 71ed3735-b4c7-47cf-aed9-7bca4ef33988 response_code = 200 uri = /v2p
36782024-08-05T16:49:45.271ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43569
36792024-08-05T16:49:45.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43569 req_id = c79ed037-c9c2-4383-9e1f-3c19446a2d63 response_code = 200 uri = /vpc-routes
36802024-08-05T16:49:45.280ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43569 req_id = d6065a67-4dbd-4133-9bcf-7660b19f41d3 response_code = 204 uri = /vpc-routes
36812024-08-05T16:49:45.361ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:32811
36822024-08-05T16:49:45.362ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:32811 req_id = ced2f485-cf6f-497e-8b9f-5b7a87ad7819 response_code = 200 uri = /vpc-routes
36832024-08-05T16:49:45.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:32811 req_id = 86978eb3-c4c2-4a62-af96-537a8f51e6d8 response_code = 204 uri = /vpc-routes
36842024-08-05T16:49:45.419ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39414
36852024-08-05T16:49:45.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39414 req_id = 90791c85-8f34-4a13-9f47-1be7c44952ff response_code = 200 uri = /v2p
36862024-08-05T16:49:45.422ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44367
36872024-08-05T16:49:45.422ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 34 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44367 req_id = 0043024f-b41a-49dc-8bee-a9d0238c7337 response_code = 200 uri = /vpc-routes
36882024-08-05T16:49:45.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44367 req_id = 94b55c49-939c-4659-bf10-52661b2f70e1 response_code = 204 uri = /vpc-routes
36892024-08-05T16:49:45.550ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51268
36902024-08-05T16:49:45.550ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51268 req_id = 4758d9c9-3fb6-4f9c-89d9-68772fc27c4b response_code = 200 uri = /v2p
36912024-08-05T16:49:45.560ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56603
36922024-08-05T16:49:45.561ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36932024-08-05T16:49:45.585ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36942024-08-05T16:49:45.608ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 48035 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:56603 req_id = 08ed2e69-9b60-45e8-adf1-0039bf560f59 response_code = 204 uri = /switch-ports
36952024-08-05T16:49:45.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 940 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56603 req_id = 56c9feeb-caee-4e51-b72a-bd9ac9811f02 response_code = 200 uri = /network-bootstore-config
36962024-08-05T16:49:45.626ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60632
36972024-08-05T16:49:45.626ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
36982024-08-05T16:49:45.650ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
36992024-08-05T16:49:45.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 43390 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60632 req_id = 20e15a2e-d558-4f4e-b9e8-70e7f4eeaa87 response_code = 204 uri = /switch-ports
37002024-08-05T16:49:45.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60632 req_id = fe335a10-81be-4969-937a-78af8aab0e6c response_code = 200 uri = /network-bootstore-config
37012024-08-05T16:49:45.685ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56495
37022024-08-05T16:49:45.685ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37032024-08-05T16:49:45.709ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37042024-08-05T16:49:45.734ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 49019 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56495 req_id = 79c2795a-a0fa-4792-82fb-f9ad23e5f048 response_code = 204 uri = /switch-ports
37052024-08-05T16:49:45.734ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56495 req_id = f2ef4bab-2495-44ba-8c87-80f077c85681 response_code = 200 uri = /network-bootstore-config
37062024-08-05T16:49:57.723ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 357 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = 6b76b066-1f35-4798-ab97-be06703e7e7a response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
37072024-08-05T16:50:01.194ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62492
37082024-08-05T16:50:01.194ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 108 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62492 req_id = c52cd505-74a9-4602-bd8c-3a0154c6e629 response_code = 200 uri = /baseboard
37092024-08-05T16:50:15.096ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59447
37102024-08-05T16:50:15.096ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59447 req_id = 2cc95b78-63af-4bb9-8cc3-877dc9341a55 response_code = 200 uri = /vpc-routes
37112024-08-05T16:50:15.103ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59447 req_id = 78008d40-6051-404b-8bbc-0b70d51a1119 response_code = 204 uri = /vpc-routes
37122024-08-05T16:50:15.108ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46299
37132024-08-05T16:50:15.109ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46299 req_id = 5b67027f-6b59-48dd-ba59-9ea252d4cb1c response_code = 200 uri = /v2p
37142024-08-05T16:50:15.125ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35942
37152024-08-05T16:50:15.126ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35942 req_id = f9f2ae37-c565-4927-907c-c799ccf36276 response_code = 200 uri = /vpc-routes
37162024-08-05T16:50:15.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:35942 req_id = fed3e4db-24fd-41c9-9e9e-ae8206e8423f response_code = 204 uri = /vpc-routes
37172024-08-05T16:50:15.144ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58188
37182024-08-05T16:50:15.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 921 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58188 req_id = 7b930c13-4099-497e-b82c-4c55180be386 response_code = 200 uri = /vpc-routes
37192024-08-05T16:50:15.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58188 req_id = d3aeb3f2-5c04-4f17-82a1-5af3018fe51e response_code = 204 uri = /vpc-routes
37202024-08-05T16:50:15.153ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37487
37212024-08-05T16:50:15.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37487 req_id = e2844057-7da9-4a33-a799-96a6d687dc82 response_code = 200 uri = /v2p
37222024-08-05T16:50:15.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52238
37232024-08-05T16:50:15.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52238 req_id = 351994a2-11bf-4688-a0d2-f798b0eb2b3e response_code = 200 uri = /v2p
37242024-08-05T16:50:15.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39474
37252024-08-05T16:50:15.260ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37262024-08-05T16:50:15.286ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37272024-08-05T16:50:15.310ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 49882 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39474 req_id = 318b6098-61ca-4642-87aa-993ad7a07710 response_code = 204 uri = /switch-ports
37282024-08-05T16:50:15.310ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44678
37292024-08-05T16:50:15.310ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48544
37302024-08-05T16:50:15.310ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 285 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39474 req_id = a9957078-ceeb-4502-9dc9-29b1646720b7 response_code = 200 uri = /network-bootstore-config
37312024-08-05T16:50:15.310ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37322024-08-05T16:50:15.334ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37332024-08-05T16:50:15.352ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37342024-08-05T16:50:15.352ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 41863 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44678 req_id = 7982c971-0909-443b-9ff1-8f6439e4a568 response_code = 204 uri = /switch-ports
37352024-08-05T16:50:15.355ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 637 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44678 req_id = 28e49e5a-1a5d-42ba-93e8-9428c0fd8886 response_code = 200 uri = /network-bootstore-config
37362024-08-05T16:50:15.376ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37372024-08-05T16:50:15.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 92041 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48544 req_id = 1f603a66-e915-4a68-a178-5353595a2792 response_code = 204 uri = /switch-ports
37382024-08-05T16:50:15.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48544 req_id = 3afd9d29-5ea9-4199-a4c2-d1695484f1af response_code = 200 uri = /network-bootstore-config
37392024-08-05T16:50:27.763ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 511 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = f4489420-fd5d-4672-9fd3-fcf0672f653a response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
37402024-08-05T16:50:31.226ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57624
37412024-08-05T16:50:31.226ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 41 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57624 req_id = 4c854913-b569-4042-9292-10e249f1ecc5 response_code = 200 uri = /baseboard
37422024-08-05T16:50:45.194ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34487
37432024-08-05T16:50:45.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34487 req_id = 1f59901e-1078-4221-8df2-ef64d9ec0853 response_code = 200 uri = /vpc-routes
37442024-08-05T16:50:45.203ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:34487 req_id = 3e9800f3-8c25-429b-ab5f-25e62842ac94 response_code = 204 uri = /vpc-routes
37452024-08-05T16:50:45.220ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33950
37462024-08-05T16:50:45.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1385 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33950 req_id = 855925a6-7f73-499c-aab8-e375dcb0c77c response_code = 200 uri = /vpc-routes
37472024-08-05T16:50:45.229ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51602
37482024-08-05T16:50:45.229ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51602 req_id = c4047efb-0587-40d5-bc73-ff5b7cfed7e6 response_code = 200 uri = /vpc-routes
37492024-08-05T16:50:45.229ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33950 req_id = 2ff7bec9-2ffb-4761-86c5-5e713e4f0150 response_code = 204 uri = /vpc-routes
37502024-08-05T16:50:45.237ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51602 req_id = 9eff9528-a3e5-4db4-a8b0-c1dc72a82b45 response_code = 204 uri = /vpc-routes
37512024-08-05T16:50:45.358ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41499
37522024-08-05T16:50:45.358ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41499 req_id = 2b42f255-5679-4f73-8786-f4eb488b0433 response_code = 200 uri = /v2p
37532024-08-05T16:50:45.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35154
37542024-08-05T16:50:45.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35154 req_id = 7f946572-98c7-4a68-8925-f11af4b22cf1 response_code = 200 uri = /v2p
37552024-08-05T16:50:45.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59405
37562024-08-05T16:50:45.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59405 req_id = e35b46d2-ce23-4b4c-a967-ae4b6c543953 response_code = 200 uri = /v2p
37572024-08-05T16:50:45.574ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36724
37582024-08-05T16:50:45.575ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37592024-08-05T16:50:45.601ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37602024-08-05T16:50:45.626ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 51674 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36724 req_id = 98fbf3c0-d404-40c8-9c9f-515e48f7f8d3 response_code = 204 uri = /switch-ports
37612024-08-05T16:50:45.626ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53723
37622024-08-05T16:50:45.627ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37632024-08-05T16:50:45.630ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3578 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36724 req_id = 584dd7b0-dbf1-4a9a-9b67-f2ec18fdab7e response_code = 200 uri = /network-bootstore-config
37642024-08-05T16:50:45.633ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50157
37652024-08-05T16:50:45.650ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37662024-08-05T16:50:45.672ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37672024-08-05T16:50:45.672ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 45280 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53723 req_id = 3edd9244-3b05-43f9-a6e2-19e600033f63 response_code = 204 uri = /switch-ports
37682024-08-05T16:50:45.675ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 692 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53723 req_id = c8622fc1-7fe0-4c40-b674-1b7c84bafeb1 response_code = 200 uri = /network-bootstore-config
37692024-08-05T16:50:45.694ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37702024-08-05T16:50:45.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 83315 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50157 req_id = 3b3bc697-15cc-43e3-b6f3-8cf430ee8065 response_code = 204 uri = /switch-ports
37712024-08-05T16:50:45.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50157 req_id = 3b5c5aa4-9f49-4f45-a805-568e373ec0a3 response_code = 200 uri = /network-bootstore-config
37722024-08-05T16:50:57.763ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 575 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = c1117e33-d206-42f5-8900-65fbb7cc95f3 response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
37732024-08-05T16:51:01.258ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39343
37742024-08-05T16:51:01.258ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39343 req_id = 373e8a38-e69f-4d7a-b3b9-e2b8f26b656d response_code = 200 uri = /baseboard
37752024-08-05T16:51:15.112ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43412
37762024-08-05T16:51:15.112ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 911 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43412 req_id = de32d076-ea06-457a-b684-c1e2915960f3 response_code = 200 uri = /vpc-routes
37772024-08-05T16:51:15.116ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41687
37782024-08-05T16:51:15.116ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41687 req_id = 64320c8a-2953-4a0e-8f9f-667fe00fd284 response_code = 200 uri = /vpc-routes
37792024-08-05T16:51:15.120ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43412 req_id = 184bd984-fcb5-4182-ad75-1ce494de70f9 response_code = 204 uri = /vpc-routes
37802024-08-05T16:51:15.121ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54240
37812024-08-05T16:51:15.121ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 29 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54240 req_id = 67421fc7-2616-49ca-a3db-99f6e895fdb7 response_code = 200 uri = /vpc-routes
37822024-08-05T16:51:15.124ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 882 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:41687 req_id = 49b0ecb5-c946-4f56-8de7-7f4773001af5 response_code = 204 uri = /vpc-routes
37832024-08-05T16:51:15.128ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54240 req_id = 21bc8d8d-06f4-42c4-a9bc-170b4f93ae57 response_code = 204 uri = /vpc-routes
37842024-08-05T16:51:15.169ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44828
37852024-08-05T16:51:15.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44828 req_id = 15a3a685-9302-4823-bb5d-8d65689038c3 response_code = 200 uri = /v2p
37862024-08-05T16:51:15.169ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47985
37872024-08-05T16:51:15.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47985 req_id = b2e0ba90-5862-4f14-8472-1d5e788577e1 response_code = 200 uri = /v2p
37882024-08-05T16:51:15.221ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38192
37892024-08-05T16:51:15.222ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 916 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38192 req_id = dc8e1665-9d50-4e60-b7c4-c4c12cb3db68 response_code = 200 uri = /v2p
37902024-08-05T16:51:15.280ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46123
37912024-08-05T16:51:15.280ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37922024-08-05T16:51:15.304ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37932024-08-05T16:51:15.327ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46718 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46123 req_id = ab711741-628a-4074-a7cd-577ac98ad178 response_code = 204 uri = /switch-ports
37942024-08-05T16:51:15.327ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59438
37952024-08-05T16:51:15.327ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
37962024-08-05T16:51:15.331ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3489 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46123 req_id = adc333eb-18e8-4960-9f64-26ce23beb43d response_code = 200 uri = /network-bootstore-config
37972024-08-05T16:51:15.344ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40647
37982024-08-05T16:51:15.352ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
37992024-08-05T16:51:15.374ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
38002024-08-05T16:51:15.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46559 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59438 req_id = be8627f5-2afe-4bce-906f-0c530e787652 response_code = 204 uri = /switch-ports
38012024-08-05T16:51:15.377ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 996 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59438 req_id = f08b9253-173b-4ce3-9a9a-cfe5ded4c0a8 response_code = 200 uri = /network-bootstore-config
38022024-08-05T16:51:15.396ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
38032024-08-05T16:51:15.417ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 72757 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40647 req_id = 08aeefae-5ebb-4ae3-9835-33baea7a9026 response_code = 204 uri = /switch-ports
38042024-08-05T16:51:15.417ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40647 req_id = a98ec130-b753-4c64-95ba-a5d01c3e39b8 response_code = 200 uri = /network-bootstore-config
38052024-08-05T16:51:27.764ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1180 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = 0ee781b4-6510-4bc3-b114-c19ae0d32d20 response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
38062024-08-05T16:51:31.291ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59980
38072024-08-05T16:51:31.291ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59980 req_id = f1a3f3f8-d81f-4fba-ad8f-b1e3ae865a22 response_code = 200 uri = /baseboard
38082024-08-05T16:51:45.146ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52432
38092024-08-05T16:51:45.146ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52432 req_id = ea5ccb34-1522-461b-8d38-4d625e9f2ea8 response_code = 200 uri = /vpc-routes
38102024-08-05T16:51:45.153ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:52432 req_id = 7cecaa86-711d-42c7-b835-2ae07dc6ba85 response_code = 204 uri = /vpc-routes
38112024-08-05T16:51:45.199ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47375
38122024-08-05T16:51:45.199ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47375 req_id = 69e6d542-3cd3-4e4f-a346-419eadc71279 response_code = 200 uri = /v2p
38132024-08-05T16:51:45.251ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49483
38142024-08-05T16:51:45.251ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49483 req_id = 09c888e6-7d2a-4a80-9c7e-545359d0f4a8 response_code = 200 uri = /vpc-routes
38152024-08-05T16:51:45.259ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49483 req_id = 500dcdc8-7a80-44c2-9343-193e92db65d1 response_code = 204 uri = /vpc-routes
38162024-08-05T16:51:45.278ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52293
38172024-08-05T16:51:45.278ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52293 req_id = f8189ec4-3e00-44cd-8fad-42aa153e3d2b response_code = 200 uri = /v2p
38182024-08-05T16:51:45.322ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37999
38192024-08-05T16:51:45.322ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37999 req_id = 997597ff-ad70-4ff3-8b54-c68543b28670 response_code = 200 uri = /v2p
38202024-08-05T16:51:45.325ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63735
38212024-08-05T16:51:45.325ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 32 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63735 req_id = 4ee94b9c-be52-45df-92fb-4a7abc80e916 response_code = 200 uri = /vpc-routes
38222024-08-05T16:51:45.331ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63735 req_id = f5ec2b35-9e72-45ad-8fc6-46ed86cd5bb6 response_code = 204 uri = /vpc-routes
38232024-08-05T16:51:45.381ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64464
38242024-08-05T16:51:45.381ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
38252024-08-05T16:51:45.381ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
38262024-08-05T16:51:45.381ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38272024-08-05T16:51:45.381ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38282024-08-05T16:51:45.381ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
38292024-08-05T16:51:45.381ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38302024-08-05T16:51:45.381ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
38312024-08-05T16:51:45.381ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 269 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64464 req_id = 113e869f-3114-4578-9b74-6af2387f2e96 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
38322024-08-05T16:51:45.512ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63585
38332024-08-05T16:51:45.512ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
38342024-08-05T16:51:45.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
38352024-08-05T16:51:45.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38362024-08-05T16:51:45.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38372024-08-05T16:51:45.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
38382024-08-05T16:51:45.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38392024-08-05T16:51:45.512ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
38402024-08-05T16:51:45.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 202 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63585 req_id = ade02128-5a44-4a1a-ad51-6d012a66ecb6 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
38412024-08-05T16:51:45.536ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39198
38422024-08-05T16:51:45.536ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:673 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0a07869d-c9ec-4ade-b114-97ceff54b60e, kind: Service { id: 8a344ff3-4023-4f5e-a29a-3db2dc6980b8 }, name: Name("external-dns-8a344ff3-4023-4f5e-a29a-3db2dc6980b8"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 206, 204])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c8555f5b-90c8-4b5b-b73e-5b0ba7ce0695, kind: Service { id: 8995285b-5cb0-4d3a-b021-8aec05314a65 }, name: Name("external-dns-8995285b-5cb0-4d3a-b021-8aec05314a65"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 233, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 38978e97-2109-4173-8624-98de2b729958, kind: Service { id: 4952ec5d-23ba-48ea-9509-011fcb02adaf }, name: Name("nexus-4952ec5d-23ba-48ea-9509-011fcb02adaf"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 216, 188])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 51730dfc-654f-49d5-a0b7-5f1b747fb3be, kind: Service { id: 1c82ddd6-6f5a-4a92-a9b3-fde01c046c42 }, name: Name("nexus-1c82ddd6-6f5a-4a92-a9b3-fde01c046c42"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 202, 28])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 6eedd1d7-7b63-4139-96c3-ad58e9233e7c, kind: Service { id: 866d6328-4ca1-435e-8a98-cd5268df29e8 }, name: Name("nexus-866d6328-4ca1-435e-8a98-cd5268df29e8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 210, 250])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
38432024-08-05T16:51:45.536ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
38442024-08-05T16:51:45.536ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38452024-08-05T16:51:45.536ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38462024-08-05T16:51:45.536ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte0" rules = []
38472024-08-05T16:51:45.536ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
38482024-08-05T16:51:45.536ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:692 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
38492024-08-05T16:51:45.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39198 req_id = 5c0694a6-44eb-4929-a37c-2156448f5f9b response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
38502024-08-05T16:51:45.574ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54075
38512024-08-05T16:51:45.574ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
38522024-08-05T16:51:45.600ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
38532024-08-05T16:51:45.623ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 48803 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54075 req_id = 07b628d3-2fd4-49ea-9cf7-557419ca53a4 response_code = 204 uri = /switch-ports
38542024-08-05T16:51:45.623ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43056
38552024-08-05T16:51:45.623ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
38562024-08-05T16:51:45.626ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3066 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54075 req_id = 1a57c281-5ecc-4646-beb0-043fd413fc8b response_code = 200 uri = /network-bootstore-config
38572024-08-05T16:51:45.647ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
38582024-08-05T16:51:45.669ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46386 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43056 req_id = 2c5bb108-48f9-4427-a70f-3e4332297eb7 response_code = 204 uri = /switch-ports
38592024-08-05T16:51:45.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43056 req_id = 7c47232a-a8c2-4d0f-bfba-a4ffd02000bb response_code = 200 uri = /network-bootstore-config
38602024-08-05T16:51:45.740ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58619
38612024-08-05T16:51:45.740ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
38622024-08-05T16:51:45.762ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
38632024-08-05T16:51:45.784ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 43487 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58619 req_id = d68103ec-eaa8-4560-8490-b495bfffab96 response_code = 204 uri = /switch-ports
38642024-08-05T16:51:45.785ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 707 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58619 req_id = 31b13999-8944-4d8b-ad8a-5bbe696328fd response_code = 200 uri = /network-bootstore-config
38652024-08-05T16:51:57.764ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1283 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = f2ed86a5-e24f-4abe-9b12-3235ad44741c response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
38662024-08-05T16:52:01.325ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35291
38672024-08-05T16:52:01.325ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:35291 req_id = ee1df76b-a09f-407c-92a8-44fbac2bd13c response_code = 200 uri = /baseboard
38682024-08-05T16:52:15.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63927
38692024-08-05T16:52:15.101ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63927 req_id = 33230b50-bc36-4d3b-b574-a1a9963fe276 response_code = 200 uri = /vpc-routes
38702024-08-05T16:52:15.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1083 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63927 req_id = 9874bffc-efba-4ccc-b7e7-d5c70801e3ca response_code = 204 uri = /vpc-routes
38712024-08-05T16:52:15.110ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39868
38722024-08-05T16:52:15.110ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 31 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39868 req_id = e913a43b-ab19-4f62-b1db-0477d808a5ee response_code = 200 uri = /vpc-routes
38732024-08-05T16:52:15.117ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39868 req_id = 759a84bf-1854-4a5a-91ef-740d5ef76573 response_code = 204 uri = /vpc-routes
38742024-08-05T16:52:15.132ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43768
38752024-08-05T16:52:15.132ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43768 req_id = c46d4624-57e5-478f-8797-7eeaaca70913 response_code = 200 uri = /v2p
38762024-08-05T16:52:15.152ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36683
38772024-08-05T16:52:15.152ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36683 req_id = f15b3f8a-21cf-4efb-a0aa-add6d8e90c50 response_code = 200 uri = /vpc-routes
38782024-08-05T16:52:15.159ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36683 req_id = 3862147f-b2b2-4828-8116-32e615a45235 response_code = 204 uri = /vpc-routes
38792024-08-05T16:52:15.192ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35145
38802024-08-05T16:52:15.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 845 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35145 req_id = f4b2c02c-53ba-44d7-85c9-4721bdaed34b response_code = 200 uri = /v2p
38812024-08-05T16:52:15.200ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64736
38822024-08-05T16:52:15.200ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64736 req_id = e66d72cb-a015-4a20-b413-f356a6ed8013 response_code = 200 uri = /v2p
38832024-08-05T16:52:15.255ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47446
38842024-08-05T16:52:15.256ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
38852024-08-05T16:52:15.280ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
38862024-08-05T16:52:15.304ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 48978 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47446 req_id = 3124e846-c835-4980-a7c7-92f622deea93 response_code = 204 uri = /switch-ports
38872024-08-05T16:52:15.304ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35278
38882024-08-05T16:52:15.304ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
38892024-08-05T16:52:15.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3502 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47446 req_id = 69fc20b8-ae6c-4438-ab7b-256b12cdeafa response_code = 200 uri = /network-bootstore-config
38902024-08-05T16:52:15.328ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
38912024-08-05T16:52:15.351ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46638 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35278 req_id = 482ee68c-ed03-41bb-915f-f8827b6f74ae response_code = 204 uri = /switch-ports
38922024-08-05T16:52:15.352ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 734 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35278 req_id = 1138e5b8-2e37-4a47-be87-68d8aea0e730 response_code = 200 uri = /network-bootstore-config
38932024-08-05T16:52:15.373ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40548
38942024-08-05T16:52:15.374ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
38952024-08-05T16:52:15.395ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
38962024-08-05T16:52:15.418ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 44043 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40548 req_id = a37d1298-4b00-43f4-bbbd-c7d90bdf6001 response_code = 204 uri = /switch-ports
38972024-08-05T16:52:15.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 904 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40548 req_id = 591b35b3-bc2b-4870-800e-c3d6ed8bb929 response_code = 200 uri = /network-bootstore-config
38982024-08-05T16:52:27.764ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 537 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = cfe2443e-cf37-4a2a-9a4f-dbf6a2be8f70 response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
38992024-08-05T16:52:31.359ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39409
39002024-08-05T16:52:31.359ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 86 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39409 req_id = 3b108dc9-72d8-4466-bc83-6c55ed6df27d response_code = 200 uri = /baseboard
39012024-08-05T16:52:33.455ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:303
39022024-08-05T16:52:33.456ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
39032024-08-05T16:52:33.493ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:292 next_cleanup = Instant { tv_sec: 12090, tv_nsec: 850417773 } time_to_next_cleanup = 599.99999578s
39042024-08-05T16:52:45.186ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45629
39052024-08-05T16:52:45.187ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1681 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45629 req_id = 035ef1f2-7bae-49c3-9a79-8ceaad0caa06 response_code = 200 uri = /vpc-routes
39062024-08-05T16:52:45.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45629 req_id = e24af0a8-2f68-4b7b-8ec3-0f4d16af220a response_code = 204 uri = /vpc-routes
39072024-08-05T16:52:45.234ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48904
39082024-08-05T16:52:45.234ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48904 req_id = cd1df7ca-a5e6-410e-8274-c3dcd934071c response_code = 200 uri = /v2p
39092024-08-05T16:52:45.360ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39334
39102024-08-05T16:52:45.361ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 24 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39334 req_id = bf75c0c2-e5f7-49eb-9777-8b54d532fcde response_code = 200 uri = /vpc-routes
39112024-08-05T16:52:45.368ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 78 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39334 req_id = 92efe327-3be2-4492-94e6-6160ff1d39bf response_code = 204 uri = /vpc-routes
39122024-08-05T16:52:45.373ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47400
39132024-08-05T16:52:45.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 947 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47400 req_id = 3dfbab9e-a5b9-4d7a-afd8-6ab7a72c6cbf response_code = 200 uri = /vpc-routes
39142024-08-05T16:52:45.381ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47400 req_id = eee97850-da2d-490b-befa-6f19db620aa5 response_code = 204 uri = /vpc-routes
39152024-08-05T16:52:45.400ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46676
39162024-08-05T16:52:45.400ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46676 req_id = e3de0d0a-5651-4402-b566-d963d49cd7bf response_code = 200 uri = /v2p
39172024-08-05T16:52:45.488ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44772
39182024-08-05T16:52:45.489ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
39192024-08-05T16:52:45.512ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
39202024-08-05T16:52:45.535ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46166 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44772 req_id = 9d6fa586-756f-4a02-806a-e9a3892ae54b response_code = 204 uri = /switch-ports
39212024-08-05T16:52:45.535ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40397
39222024-08-05T16:52:45.535ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37731
39232024-08-05T16:52:45.535ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 204 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44772 req_id = 7676728f-e7fe-4e52-b6b1-0607ad48734b response_code = 200 uri = /network-bootstore-config
39242024-08-05T16:52:45.535ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
39252024-08-05T16:52:45.535ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 304 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37731 req_id = cd6bbcec-07b4-4020-8806-8b8b6b4e5330 response_code = 200 uri = /v2p
39262024-08-05T16:52:45.558ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
39272024-08-05T16:52:45.578ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 43306 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40397 req_id = 92bb5965-7799-496a-9e6e-7e692e848e2f response_code = 204 uri = /switch-ports
39282024-08-05T16:52:45.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 854 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40397 req_id = 9b98ad89-3d8b-4eea-b58a-ed99b2db0e36 response_code = 200 uri = /network-bootstore-config
39292024-08-05T16:52:45.683ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35560
39302024-08-05T16:52:45.683ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
39312024-08-05T16:52:45.705ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
39322024-08-05T16:52:45.726ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 43365 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35560 req_id = b18f514a-9ce7-4794-abf4-d95ea7b9ea2f response_code = 204 uri = /switch-ports
39332024-08-05T16:52:45.727ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35560 req_id = 531e0cd4-8a30-4304-823d-4f8d06c2d481 response_code = 200 uri = /network-bootstore-config
39342024-08-05T16:52:47.249ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
39352024-08-05T16:52:57.841ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 1431 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = 4b6ee169-f9aa-47ad-b6be-c1b25ea78c80 response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
39362024-08-05T16:53:01.392ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55782
39372024-08-05T16:53:01.393ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 630 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:55782 req_id = 9f1a3c38-3675-4d01-8eec-0475ee26e7a6 response_code = 200 uri = /baseboard
39382024-08-05T16:53:15.121ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39366
39392024-08-05T16:53:15.121ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39366 req_id = 5b266bdd-8cd6-42cf-bca5-7016c1518aca response_code = 200 uri = /vpc-routes
39402024-08-05T16:53:15.129ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39366 req_id = 0e56fdfa-5323-4ab8-8447-15b30cc70186 response_code = 204 uri = /vpc-routes
39412024-08-05T16:53:15.133ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48109
39422024-08-05T16:53:15.133ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48109 req_id = fc55e361-4845-4a32-9a26-3f0612edc147 response_code = 200 uri = /vpc-routes
39432024-08-05T16:53:15.140ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55103
39442024-08-05T16:53:15.140ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55103 req_id = 13ff5be7-018a-41a5-9339-d00f765d6ff7 response_code = 200 uri = /vpc-routes
39452024-08-05T16:53:15.141ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48109 req_id = 24c986d9-9a7f-45fe-a71f-b6403c297eb7 response_code = 204 uri = /vpc-routes
39462024-08-05T16:53:15.147ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55103 req_id = a4b88122-a0b8-4e6b-98a0-6a009235351e response_code = 204 uri = /vpc-routes
39472024-08-05T16:53:15.155ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52734
39482024-08-05T16:53:15.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52734 req_id = eb4d0b71-9269-4268-84a0-7d148142cd6c response_code = 200 uri = /v2p
39492024-08-05T16:53:15.174ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35376
39502024-08-05T16:53:15.174ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35376 req_id = 4acf2b60-913a-49bc-85a2-acf5c0716a98 response_code = 200 uri = /v2p
39512024-08-05T16:53:15.215ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37537
39522024-08-05T16:53:15.215ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37537 req_id = dc076068-5d2f-42ba-af9e-a645aa526403 response_code = 200 uri = /v2p
39532024-08-05T16:53:15.287ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53817
39542024-08-05T16:53:15.287ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
39552024-08-05T16:53:15.309ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
39562024-08-05T16:53:15.331ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 43827 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:53817 req_id = cefa6018-29f3-4b25-9c7d-6b5044215d60 response_code = 204 uri = /switch-ports
39572024-08-05T16:53:15.331ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61216
39582024-08-05T16:53:15.331ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
39592024-08-05T16:53:15.335ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3192 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53817 req_id = b14d4960-d286-46c3-8ae0-3dd48b41f2a1 response_code = 200 uri = /network-bootstore-config
39602024-08-05T16:53:15.353ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42554
39612024-08-05T16:53:15.355ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
39622024-08-05T16:53:15.376ZINFOSledAgent (ServiceManager): Setting up uplinkd service file = sled-agent/src/services.rs:3823
39632024-08-05T16:53:15.376ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 44527 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61216 req_id = 978250ab-c604-475e-8aa1-c7fecab7b579 response_code = 204 uri = /switch-ports
39642024-08-05T16:53:15.379ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 517 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61216 req_id = 4b4f838f-5bf6-47b2-991a-87eabf418e95 response_code = 200 uri = /network-bootstore-config
39652024-08-05T16:53:15.400ZINFOSledAgent (ServiceManager): configuring port: HostPortConfig { port: "qsfp0", addrs: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }] } file = sled-agent/src/services.rs:3834
39662024-08-05T16:53:15.422ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 68941 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:42554 req_id = 2bcf1f37-d715-42c8-b0f5-81aa1a1df064 response_code = 204 uri = /switch-ports
39672024-08-05T16:53:15.423ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42554 req_id = 40e83b8d-bcdd-48e3-b50d-9310d04d5972 response_code = 200 uri = /network-bootstore-config
39682024-08-05T16:53:18.940ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63368
39692024-08-05T16:53:18.941ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 787 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63368 req_id = 26dc6adb-6719-4fd2-a7a9-55f905cafa70 response_code = 200 uri = /v2p
39702024-08-05T16:53:18.944ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:780 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(8395431) }
39712024-08-05T16:53:18.945ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 3146 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63368 req_id = 7de677ac-08ae-46a1-8168-d018b3261577 response_code = 204 uri = /v2p
39722024-08-05T16:53:19.066ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63491
39732024-08-05T16:53:19.073ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:592 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 7e30d6a5-06c5-48b0-a304-819497336a2b, kind: Instance { id: 60194fb6-a280-4030-a452-3f4a705fbb54 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(8395431), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 7e30d6a5-06c5-48b0-a304-819497336a2b, kind: Instance { id: 60194fb6-a280-4030-a452-3f4a705fbb54 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(8395431), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 7e30d6a5-06c5-48b0-a304-819497336a2b, kind: Instance { id: 60194fb6-a280-4030-a452-3f4a705fbb54 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(8395431), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(8395431))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 7e30d6a5-06c5-48b0-a304-819497336a2b, kind: Instance { id: 60194fb6-a280-4030-a452-3f4a705fbb54 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(8395431), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-95063f50cc0", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 3b001efd-ef58-42b1-9232-aa581e636831, read_only_parent: Some(Volume { block_size: 512, id: 9f47f0e9-08aa-463b-8505-23feaed489ca, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4845401b-1c93-4371-965c-ef447b2b4c4e, key: Some("wBLcPlGM1lU0a4cNONAjXCEYfJYa8e7+N/aJipdWaCw="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 3b001efd-ef58-42b1-9232-aa581e636831, key: Some("a4ZH8kEQG4WXhxpUzSemMPaL2KbplBadaQMt3A1Xox8="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::14]:19002", "[fd00:1122:3344:101::19]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABnqYYFWQVZAACphgVZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAZ6mGBVkFWQAAqYYFWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI2MDE5NGZiNi1hMjgwLTQwMzAtYTQ1Mi0zZjRhNzA1ZmJiNTQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJQlJ3VnF1TjY1Z0lCYWwwc1NXUzBaQ01ia3dVRlJxY0tlNGFVVW1DZGRYMSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 instance_runtime = InstanceRuntimeState { propolis_id: Some(a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-05T16:53:18.731554843Z } metadata = InstanceMetadata { silo_id: e179a288-a562-4f81-9527-488ea538b84b, project_id: 19b9bd76-f1e4-448f-b46c-d56eb60fe928 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = a99b853a-2530-488f-b74e-e3781d0d4d3c vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-08-05T16:53:18.684161Z }
39742024-08-05T16:53:19.073ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:626 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 (instance)
39752024-08-05T16:53:19.073ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:992 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 propolis_id = a99b853a-2530-488f-b74e-e3781d0d4d3c state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 7e30d6a5-06c5-48b0-a304-819497336a2b, kind: Instance { id: 60194fb6-a280-4030-a452-3f4a705fbb54 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(8395431), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 7e30d6a5-06c5-48b0-a304-819497336a2b, kind: Instance { id: 60194fb6-a280-4030-a452-3f4a705fbb54 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(8395431), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 7e30d6a5-06c5-48b0-a304-819497336a2b, kind: Instance { id: 60194fb6-a280-4030-a452-3f4a705fbb54 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(8395431), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(8395431))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 7e30d6a5-06c5-48b0-a304-819497336a2b, kind: Instance { id: 60194fb6-a280-4030-a452-3f4a705fbb54 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(8395431), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-95063f50cc0", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 3b001efd-ef58-42b1-9232-aa581e636831, read_only_parent: Some(Volume { block_size: 512, id: 9f47f0e9-08aa-463b-8505-23feaed489ca, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4845401b-1c93-4371-965c-ef447b2b4c4e, key: Some("wBLcPlGM1lU0a4cNONAjXCEYfJYa8e7+N/aJipdWaCw="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 3b001efd-ef58-42b1-9232-aa581e636831, key: Some("a4ZH8kEQG4WXhxpUzSemMPaL2KbplBadaQMt3A1Xox8="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::14]:19002", "[fd00:1122:3344:101::19]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABnqYYFWQVZAACphgVZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAZ6mGBVkFWQAAqYYFWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI2MDE5NGZiNi1hMjgwLTQwMzAtYTQ1Mi0zZjRhNzA1ZmJiNTQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJQlJ3VnF1TjY1Z0lCYWwwc1NXUzBaQ01ia3dVRlJxY0tlNGFVVW1DZGRYMSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-05T16:53:18.731554843Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-08-05T16:53:18.684161Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
39762024-08-05T16:53:19.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 7508 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63491 req_id = 1045f688-f8f0-45b0-8aef-cd17816ef2cd response_code = 200 uri = /instances/60194fb6-a280-4030-a452-3f4a705fbb54
39772024-08-05T16:53:19.208ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37587
39782024-08-05T16:53:19.211ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 2499 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37587 req_id = 5b98ab56-f6e5-46f1-94b7-8b4b83a2f6c3 response_code = 200 uri = /instances/60194fb6-a280-4030-a452-3f4a705fbb54/state
39792024-08-05T16:53:19.216ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:426 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 244, 103, 212]), slot: 0, vni: Vni { inner: 8395431 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
39802024-08-05T16:53:19.216ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
39812024-08-05T16:53:19.252ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c file = illumos-utils/src/zone.rs:316 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39822024-08-05T16:53:19.274ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c file = illumos-utils/src/zone.rs:354 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39832024-08-05T16:53:20.636ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39842024-08-05T16:53:20.636ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c zone file = illumos-utils/src/running_zone.rs:472 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39852024-08-05T16:53:20.815ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 51.567ms file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39862024-08-05T16:53:20.876ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 137.460513ms file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39872024-08-05T16:53:21.038ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 229.777213ms file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39882024-08-05T16:53:21.282ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 404.130623ms file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39892024-08-05T16:53:21.700ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 1.077818409s file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39902024-08-05T16:53:22.791ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 714.956478ms file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39912024-08-05T16:53:23.515ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 1.431609109s file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39922024-08-05T16:53:24.957ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 1.118144274s file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 zone = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
39932024-08-05T16:53:26.105ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c file = sled-agent/src/instance.rs:1490 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39942024-08-05T16:53:26.114ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 42.655881ms file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39952024-08-05T16:53:26.166ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 129.250035ms file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39962024-08-05T16:53:26.306ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c") failed: Property not found. retry in 213.493604ms file = illumos-utils/src/svc.rs:36 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39972024-08-05T16:53:26.530ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1499 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39982024-08-05T16:53:26.564ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1513 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
39992024-08-05T16:53:26.565ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAABnqYYFWQVZAACphgVZAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAZ6mGBVkFWQAAqYYFWQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiI2MDE5NGZiNi1hMjgwLTQwMzAtYTQ1Mi0zZjRhNzA1ZmJiNTQiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJQlJ3VnF1TjY1Z0lCYWwwc1NXUzBaQ01ia3dVRlJxY0tlNGFVVW1DZGRYMSJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-95063f50cc0", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 3b001efd-ef58-42b1-9232-aa581e636831, read_only_parent: Some(Volume { block_size: 512, id: 9f47f0e9-08aa-463b-8505-23feaed489ca, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 4845401b-1c93-4371-965c-ef447b2b4c4e, key: Some("wBLcPlGM1lU0a4cNONAjXCEYfJYa8e7+N/aJipdWaCw="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 3b001efd-ef58-42b1-9232-aa581e636831, key: Some("a4ZH8kEQG4WXhxpUzSemMPaL2KbplBadaQMt3A1Xox8="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19002", "[fd00:1122:3344:101::14]:19002", "[fd00:1122:3344:101::19]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "opte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: 60194fb6-a280-4030-a452-3f4a705fbb54, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 19b9bd76-f1e4-448f-b46c-d56eb60fe928, silo_id: e179a288-a562-4f81-9527-488ea538b84b, sled_id: 3abf3a4f-8820-40af-a01f-4369b42a5529, sled_model: "i86pc", sled_revision: 0, sled_serial: "buskin" }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:736 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40002024-08-05T16:53:26.704ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:738 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40012024-08-05T16:53:26.706ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_status: NoMigration, time: 2024-08-05T16:53:26.706377216Z } file = sled-agent/src/instance.rs:622 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40022024-08-05T16:53:26.706ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:644 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 new_instance_state = InstanceRuntimeState { propolis_id: Some(a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-05T16:53:18.731554843Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-08-05T16:53:26.706377216Z } propolis_id = a99b853a-2530-488f-b74e-e3781d0d4d3c
40032024-08-05T16:53:26.706ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:541 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-05T16:53:18.731554843Z }, propolis_id: a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis), vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-08-05T16:53:26.706377216Z }, migration_state: None }
40042024-08-05T16:53:26.774ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-08-05T16:53:26.774570929Z } file = sled-agent/src/instance.rs:622 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40052024-08-05T16:53:26.774ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:644 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 new_instance_state = InstanceRuntimeState { propolis_id: Some(a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-05T16:53:18.731554843Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-08-05T16:53:26.774570929Z } propolis_id = a99b853a-2530-488f-b74e-e3781d0d4d3c
40062024-08-05T16:53:26.774ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:541 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-05T16:53:18.731554843Z }, propolis_id: a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis), vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-08-05T16:53:26.774570929Z }, migration_state: None }
40072024-08-05T16:53:26.810ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52536
40082024-08-05T16:53:26.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 698 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52536 req_id = 0c823a25-c65c-4202-b18f-370cc1384b70 response_code = 200 uri = /vpc-routes
40092024-08-05T16:53:26.854ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 242 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52536 req_id = dba5c045-d8b1-4638-995b-39ce1418bc5d response_code = 204 uri = /vpc-routes
40102024-08-05T16:53:26.892ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56208
40112024-08-05T16:53:26.892ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56208 req_id = f083fbdf-6fa1-4488-95e9-c47bae861893 response_code = 200 uri = /vpc-routes
40122024-08-05T16:53:26.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56208 req_id = 0cbc9130-e917-40de-aba0-94fcf4104763 response_code = 204 uri = /vpc-routes
40132024-08-05T16:53:27.841ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 534 local_addr = [fd00:1122:3344:101::1]:40509 method = GET remote_addr = [fd00:1122:3344:101::d]:56611 req_id = e2454d12-d00e-4903-8199-bd7df9a6bad1 response_code = 200 sled_id = 3abf3a4f-8820-40af-a01f-4369b42a5529 uri = /3abf3a4f-8820-40af-a01f-4369b42a5529
40142024-08-05T16:53:31.427ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62740
40152024-08-05T16:53:31.427ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 83 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62740 req_id = ba8b2c3d-59fd-49b1-98c8-f0bfa43fc7f1 response_code = 200 uri = /baseboard
40162024-08-05T16:53:35.541ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59763
40172024-08-05T16:53:35.542ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 921 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59763 req_id = 66dc87f2-72e7-41c8-b414-71f6ada4d1de response_code = 200 uri = /instances/60194fb6-a280-4030-a452-3f4a705fbb54/state
40182024-08-05T16:53:35.542ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-08-05T16:53:35.542223942Z } file = sled-agent/src/instance.rs:622 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40192024-08-05T16:53:35.542ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:644 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 new_instance_state = InstanceRuntimeState { propolis_id: Some(a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-05T16:53:18.731554843Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-08-05T16:53:35.542223942Z } propolis_id = a99b853a-2530-488f-b74e-e3781d0d4d3c
40202024-08-05T16:53:35.542ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:541 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-05T16:53:18.731554843Z }, propolis_id: a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis), vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-08-05T16:53:35.542223942Z }, migration_state: None }
40212024-08-05T16:53:35.574ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-08-05T16:53:35.574476623Z } file = sled-agent/src/instance.rs:622 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40222024-08-05T16:53:35.574ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:644 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-08-05T16:53:35.574476623Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-08-05T16:53:35.574476623Z } propolis_id = a99b853a-2530-488f-b74e-e3781d0d4d3c
40232024-08-05T16:53:35.574ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:659 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40242024-08-05T16:53:35.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
40252024-08-05T16:53:35.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:405
40262024-08-05T16:53:35.574ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/4576fedf-479b-43b3-8fad-bb8b7ec274f8/debug/bundle/zone", "/pool/int/d09b3f64-0a36-4052-8233-a94e62e662e2/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/bd033efe-47fe-4ba2-8bc4-ba99e628ada7/crypt/debug", "/pool/ext/1aa678c2-e32d-4428-a115-7e698777172b/crypt/debug", "/pool/ext/cf764ad4-ee9a-4ce5-bd3d-f5a7f6798396/crypt/debug", "/pool/ext/939962e4-70b1-47be-b2d6-938c818b62c4/crypt/debug", "/pool/ext/52c33af8-e5fc-4cb6-93db-7a529f23be77/crypt/debug", "/pool/ext/cb14544a-cc03-4a04-84a7-da1d51f0a71d/crypt/debug", "/pool/ext/2cb76428-77cf-4416-ab3e-f66fc49a03b4/crypt/debug", "/pool/ext/1a42fe8d-3436-46f3-8051-0d2ff2a0be4f/crypt/debug", "/pool/ext/85b774a7-53a2-4f22-a027-1f642ba2d7f9/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:445 zone_name = oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c
40272024-08-05T16:53:35.611ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58434
40282024-08-05T16:53:35.612ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 748 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58434 req_id = c88b249d-618b-47a2-a623-9fbd9c4d026f response_code = 200 uri = /vpc-routes
40292024-08-05T16:53:35.623ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58434 req_id = d2aba864-f7d2-4c6c-b55b-01a497d78a36 response_code = 204 uri = /vpc-routes
40302024-08-05T16:53:36.397ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1249 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c", bundle_id: e4e01b87-032c-44b4-8c80-8eec8f3a2869 }, time_created: 2024-08-05T16:53:35.575014722Z, version: 0, cause: TerminatedInstance }
40312024-08-05T16:53:36.397ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_a99b853a-2530-488f-b74e-e3781d0d4d3c file = sled-agent/src/instance.rs:831 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40322024-08-05T16:53:38.169ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40332024-08-05T16:53:38.185ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:906 id = RouterId { vni: Vni(8395431), kind: System }
40342024-08-05T16:53:38.185ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:906 id = RouterId { vni: Vni(8395431), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
40352024-08-05T16:53:38.186ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:541 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-08-05T16:53:35.574476623Z }, propolis_id: a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-08-05T16:53:38.186089998Z }, migration_state: None }
40362024-08-05T16:53:38.308ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51985
40372024-08-05T16:53:38.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 242 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51985 req_id = 33e43e8e-c037-4a04-88ed-235e374e12ea response_code = 200 uri = /v2p
40382024-08-05T16:53:38.308ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:780 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(8395431) }
40392024-08-05T16:53:38.308ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51985 req_id = 2f35e687-f8c6-42c1-935f-45ffcaf075bf response_code = 204 uri = /v2p
40402024-08-05T16:53:38.672ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:541 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-08-05T16:53:35.574476623Z }, propolis_id: a99b853a-2530-488f-b74e-e3781d0d4d3c (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-08-05T16:53:38.186089998Z }, migration_state: None }
40412024-08-05T16:53:38.676ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:774 instance_id = 60194fb6-a280-4030-a452-3f4a705fbb54
40422024-08-05T16:53:38.709ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57773
40432024-08-05T16:53:38.709ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57773 req_id = 96049b85-91aa-4440-86c2-abfce036a133 response_code = 200 uri = /vpc-routes
40442024-08-05T16:53:38.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:57773 req_id = 0f2cf1b7-75c8-4d00-b626-c46a1a7a90c6 response_code = 204 uri = /vpc-routes
40452024-08-05T16:53:38.756ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58512
40462024-08-05T16:53:38.757ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58512 req_id = b9b289a9-1efc-4a3c-a34a-089b3db8230c response_code = 200 uri = /vpc-routes
40472024-08-05T16:53:38.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58512 req_id = 02eb465f-55f8-4160-99de-a62ab96c3b0e response_code = 204 uri = /vpc-routes
40482024-08-05T16:53:39.268ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34318
40492024-08-05T16:53:39.269ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34318 req_id = a18977f6-b66f-4957-ae94-83b3cb17b8e7 response_code = 200 uri = /v2p
40502024-08-05T16:53:39.269ZINFOSledAgent (PortManager): Clearing mapping of virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:820 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 244, 103, 212])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(8395431) }
40512024-08-05T16:53:39.269ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 215 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:34318 req_id = 6fb45b3a-80f4-4353-bcb4-de97a1f03c87 response_code = 204 uri = /v2p
40522024-08-05T16:53:41.346ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51735
40532024-08-05T16:53:41.346ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51735 req_id = 2471c248-3ba5-4df0-a241-bad3bf9a4467 response_code = 200 uri = /vpc-routes
40542024-08-05T16:53:41.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0c0bae3/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51735 req_id = 1d65cc02-c9f6-4272-b0ad-a9aaf3fcc039 response_code = 204 uri = /vpc-routes